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Introduction 

Appellants filed the present patent application on March 7, 2000. On March 7, 2001 a 
petition to make special was filed by Appellants based on actual infringement pursuant to 37 
C.F.R. § 1 .102(d). A first office action was issued on July 6, 2001 rejecting all pending claims 
based on a combination of references. Appellants' attorney conducted a personal interview of 
the examiner and the primary examiner on July 30, 2001 during which the examiners suggested 
that certain claim amendments be made to overcome the prior art. These amendments were 
made and submitted in a formal reply on September 27, 2001 . On December 26, 2001, an office 
action was issued that rejected all of the pending claims based on a new combination of 
references. On March 26, 2002, Appellants filed a reply presenting arguments in favor of 
patentability. On June 20, 2002, an office action was issued that rejected all of the pending 
claims based on a new combination of references. 

Accordingly, on July 26, 2002 Appellants' attorney conducted a telephonic interview 
with the examiner and the primary examiner during which an agreement on the claims was 
reached after a discussion of the differences between the prior art and Appellants' invention. On 
September 20, 2002 Appellants submitted an amendment pursuant to the discussion and 
agreement during the interview. On December 18, 2002, the Final Action was issued that 
rejected all of the pending claims on yet another combination of references. Appellants filed a 
notice of appeal on February 20, 2003 and hereby submit the corresponding brief in support of 
patentability of all pending claims. 

(1) Real Party in Interest 

The real party in interest is Neoplasia Press, Inc., a Michigan corporation and the 
assignee of the patent application. 

(2) Related Appeals and Interferences 

Appellants are unaware of any related appeals or interference proceedings. 

(3) Status of the Claims 

At the time of filing this appeal brief, claims 1, 3, 4, 5, 6, 8, 9, 10, 1 1, 12, 15, 16, 19, 20, 
23, and 24 are rejected. Each rejected claim is appealed. Claims 2, 7, 13, 14, 17, 18, 21, and 22 
are canceled. 



110658.000005/398628.01 



2 



110658-005 
Patent 

(4) Status of Amendments 

No amendment has been filed subsequent to the Final Action. 

(5) Summary of Invention 

The invention provides a computer implemented method for peer review of articles. The 
claimed peer review method is conducted by a central server computer over a communications 
network such as the Internet. The parties to the peer review process are the author(s), the 
editor(s), and the reviewer(s). Initially, the server receives the article from the author via the 
communications network. (Page 20, Lines 9 - 20; Fig. 2, step 110). The server computer then 
extracts contextual information from the article to determine, for example, the subject matter of 
the article. (Page 1 1, Lines 16-20; Fig. 2, steps 140, 150). The server computer then queries a 
database containing profiles of potential reviewers to identify a qualified reviewer. (Page 12, 
Lines 12-13; Fig. 3, step 200). The server computer next assigns the qualified reviewer to the 
submitted article and provides an evaluation form to the reviewer. (Page 12, Lines 16 - 20; Fig. 
3, step 230). Once the reviewer has reviewed the article and entered comments into the 
evaluation form (Page 15, Lines 12-13), the server computer receives a completed evaluation 
form from the reviewer (Page 16, Lines 13-14; Fig. 4, step 340) and provides that completed 
form to the author. (Page 17, Line 19 - Page 18, Line 3; Fig. 5, steps 400, 410, 420). Once the 
author has read the reviewer's comments and entered his own comments into the evaluation form 
(Page 18, Lines 4 - 12; Fig. 5, steps 430, 440), the server computer receives the same completed 
evaluation form with the author's comments juxtaposed with the reviewer's comments in a 
point-by-counterpoint format. (Page 18, Lines 13-16). The server computer provides this 
completed evaluation form (including the author's responses) to an editor. (Page 19, Lines 10 - 
13; Fig. 6, step 520). The server computer next receives a publication decision from the editor 
(Page 19, Lines 14-16; Fig. 6, step 530) and then provides that publication decision to the 
author and the reviewer (Page 21, Lines 5-6; Fig. 7, step 660). 

The invention also provides a separately patentable computer implemented method for 

peer review of articles. This additional claimed peer review method is also conducted by a 

central server computer over a communications network such as the Internet. The parties to the 

peer review process are the author(s), the editor(s), and the reviewer(s). Initially, the server 

receives the article from the author via the communications network. (Page 20, Lines 9 - 20; 

Fig. 2, step 110). The server computer then extracts contextual information from the article to 
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determine, for example, the subject matter of the article. (Page 11, Lines 16 - 20; Fig. 2, steps 
140, 150). The server computer then queries a database containing profiles of potential 
reviewers to identify two or more qualified reviewers. (Page 12, Lines 12-13; Fig. 3, step 200). 
All of the qualified reviewers are then ranked by the server computer. (Page 13, Lines 9-10; 
Fig. 3, step 210). The server computer then contacts the qualified reviewers and requests an 
agreement by the qualified reviewer to review the article. (Page 13, Lines 15-22; Fig. 3, step 
230). The server computer then receives agreements from one or more of the qualified reviewers 
and provides an evaluation form to each accepting reviewer. (Page 14, Lines 7-15; Fig. 3, step 
240). Once the reviewer has reviewed the article and entered comments into the evaluation form 
(Page 15, Lines 12 - 13), the server computer receives a completed evaluation form from the 
reviewer (Page 16, Lines 13-14; Fig. 4, step 340) and provides that completed form to the 
author. (Page 17, Line 19 - Page 18, Line 3; Fig. 5, steps 400, 410, 420). Once the author has 
read the reviewer's comments and entered his own comments into the evaluation form (Page 18, 
Lines 4 - 12; Fig. 5, steps 430, 440), the server computer receives the same completed evaluation 
form with the author's comments juxtaposed with the reviewer's comments in a point-by- 
counterpoint format. (Page 18, Lines 13-16). The server computer provides this completed 
evaluation form (including the author's responses) to an editor. (Page 19, Lines 10-13; Fig. 6, 
step 520). The server computer next receives a publication decision from the editor (Page 19, 
Lines 14-16; Fig. 6, step 530) and then provides that publication decision to the author and the 
reviewer (Page 21, Lines 5-6; Fig. 7, step 660). 
(6) Issues 

A. Whether claims 1, 3, 5, 6, and 8 are patentable under 35 U.S.C. 103 over the 
combination of (1) Maddison; (2) Pope; (3) Mathews; (4) Kao; (5) Sumner; (6) 
Walker; and (7) Borovoy. 

B. Whether claims 9, 10, 1 1, 12, 15, 19, 20, 23, and 24 are patentable under 35 
U.S.C. 103 over the combination of (1) Pope; (2) Maddison; (3) Mathews; (4) 
Kao; (5) Sumner; (6) Walker; and (7) Borovoy. 

C. Whether claim 4 is patentable under 35 U.S.C. 103 over the combination of (1) 
Maddison; (2) Pope; (3) Mathews; (4) Kao; (5) Sumner; (6) Walker; (7) Borovoy; 
and (8) Sato. 
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D. Whether claim 16 is patentable under 35 U.S.C. 103 over the combination of (1) 
Pope; (2) Maddison; (3) Mathews; (4) Kao; (5) Sumner; (6) Walker; (7) Borovoy; 
and (8) Sato. 

(7) Grouping of Claims 

The claims in Group A above do not stand or fall together. 
The claims in Group B above do not stand or fall together. 

(8) Argument 

A. Whether claims 1, 3, 5, 6, and 8 are patentable under 35 U.S.C. 103 over the 
combination of (1) Maddison; (2) Pope; (3) Mathews; (4) Kao; (5) Sumner; 
(6) Walker; and (7) Borovoy. 

Claims 1, 3, 5, and 8 stand or fall together. 

Independent claim 1 was rejected in the Final Action based on a combination of the 
references listed above. In particular, the rejection relies on Maddison and Mathews, which are 
cited as teaching certain steps as follows: 

Maddison -providing the completed evaluation form to the author; 

-receiving a response from the author directly in the completed evaluation form; 
-providing the completed evaluation form with author responses to an editor; 

Mathews -providing the completed evaluation form to the author; 

These references, however, standing alone or in combination fail to teach the three 
elements as demonstrated below. 
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Maddison -providing the completed evaluation form to the author; 

The Final Action cites Maddison at page 2, lines 27 - 29 and 33 1 as teaching this 
element. Specifically, Maddison states: 

Reviews are usually conducted on-line, with the reviewer typing 
comments about individual sections of a contribution into the 
windows of a structured evaluation form provided on a special 
peer-review web page. 
Maddison, Page 2, Lines 36 - 38. 

The comments furnished by the reviewers are passed along to the 
authors. 

Maddison, Page 3, Line 1 

Upon review of the cited passages, Maddison expressly teaches that the reviewer types 
comments into a structured evaluation form that is provided on a special peer-review web page. 
It is also expressly taught that those comments are passed along to the authors. What Maddison 
fails to teach is that the completed evaluation form is provided to the author, as required by claim 
1. At best, Maddison is silent with respect to teaching that the completed evaluation form is 
provided to the author because its express teachings only state that the comments are passed 
along to the author. Importantly, Maddison does not expressly teach that the completed 
evaluation form is provided to the author. 

In Ex parte Sarallo, 2002 WL 1801499 (Bd.Pat.App & Interf), the Board found that 
there was no express teaching of the subject matter asserted by the examiner to be present at the 
cited location in the reference. Ex part Sarallo, at 2. Furthermore, the Board found that the 
reference relied upon by the Examiner was silent as to the operation of the claimed element. Id. 
Maddison is equally silent as to providing the completed evaluation form to the author. In Ex 
parte Sarallo, the Board reversed the 103 rejection of the claim and stated that to interpret the 



1 Appellants have submitted copies of the non-patent references discussed in this appeal 
brief as Exhibits A - D to provide a consistent frame of reference for the page numbers and line 
numbers with respect to each reference. Appellants' review of the references with respect to the 
Examiner's citations reveal that the Examiner's copies (which have not been provided to 
Appellants) have slightly different page and line numbering than Appellants' copies. 
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cited reference as teaching the claimed element would be speculation on its part, which the 
Board stated it would not do. Id. Appellants request that the Board also refrain from speculation 
here and reverse the rejection of claim 1 . 

Additionally, in fee parte Ban, 1996 WL 1748734 (Bc.Pat.App & Interf), the Board 
found that each specification of two references were silent as to the spatial relationship of certain 
elements required by the claim, even though the figures in both references showed the elements 
arranged with spacing. Ex parte Ban, at 3. The Board stated that under the circumstances, 
where the references were silent, the examiner's position that the reference suggested the 
claimed element was not well taken. Id, at 4. Moreover, the Board further acknowledged that 
the examiner's reading of what the references suggested was based on hindsight gleaned from 
reading the Appellants disclosure rather than on the fair teachings of the references. Similarly 
here, a reading of Maddison that it suggests providing the completed evaluation form to the 
author necessarily requires the benefit of Appellants' disclosure because such a step is not 
expressly taught or fairly suggested by Maddison. Accordingly, Appellants respectfully request 
that the rejection of claim 1 based on Maddison be reversed. 

Maddison -receiving a response from the author directly in the completed evaluation form; 

The Final Action cites Maddison as teaching "receiving a response from the author 
(Maddison, page 3, lines 1 - 7, teaches that the author responses [sic] to the reviewers when the 
author decline [sic] to make the requested changes from the reviewer)." Specifically, Maddison 
states: 

The comments furnished by the reviewers are passed along to the 
authors. If the reviewers recommend that a contribution be 
revised, the authors are asked to amend the page and resubmit it to 
the editor. Authors may at this point decline to make the requested 
changed, but they would be expected to support their opinion with 
plausible arguments. Any such explanation would first be 
considered by the editor-in-chief in consultation with associate 
editors, but it might also be forwarded to the reviewers for further 
comment. 

Maddison, Page 3, Lines 1-7. 
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The cited passage here states that an author is expected to support her decision not to 
amend and resubmit the page with plausible arguments. Maddison further states that any such 
arguments would be considered by the editors and possibly the reviewers. Thus, although 
Maddison teaches that responses from the authors are received, what Maddison fails to teach is 
that the responses from the author are received directly in the completed evaluation form, as 
required by claim 1 . At best, Maddison is silent with respect to teaching that the author's 
comments are received directly in the evaluation form because its express teachings only state 
that "authors would be expected to support their opinion with plausible arguments" and that 
"[a]ny such explanation would first be considered by the editor-in-chief. . .". Importantly, 
Maddison does not expressly teach receiving a response from the author directly in the 
completed evaluation form. 

Ex parte Sarallo and Ex parte Ban are equally applicable here because this claim element 
is not expressly taught or fairly suggested by Maddison. Thus, any reading of Maddison by 
Board that it teaches or suggests receiving a response from the author directly in the completed 
evaluation form would be speculation. Moreover, similar to Ex parte Ban, the position in the 
Final Action that Maddison, although silent, teaches this step suggests that the examiner's 
reading of Maddison is based on hindsight gleaned from reading the Appellants disclosure rather 
than on the fair teachings of the reference. Accordingly, Appellants respectfully request that that 
the Board refrain from speculation and hindsight here and that the rejection of claim 1 based on 
Maddison be reversed. 

Maddison -providing the completed evaluation form with author responses to an editor; 

The Final Action cites Maddison as teaching "providing the author responses to an editor 
(Maddison, page 3, lines 1 - 7, teaches that the editors consider the author response.)" The 
language from Maddison, page 3, lines 1 - 7 is set forth in full above. 

Notably, the claim element is not merely the limitation that the author's responses are 
provide to an editor as the examiner suggests. Rather, the claimed element is providing the 
completed evaluation form with author responses to an editor. The claimed limitation is 
considerably different than the limitation rejected by the examiner and the cited passage does not 
fairly teach or suggest the claimed element. 
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As with the elements addressed above, Maddison is at best silent with respect to teaching 
the claimed element of providing the completed evaluation form with author responses to an 
editor. As explained in the specification (page 18 5 line 4 - page 19, line 4; and page 19, lines 14 

- 16), this point by counter-point manner of presenting the reviewer's comments and the author's 
responses simplifies the editor's examination of the article, the review and the corresponding 
remarks, which in turn expedites the publication recommendation and decision and thereby 
reduces the time to publication for the article. Notably, a clear advantage associated with 
providing the completed evaluation form with author responses to an editor includes an 
accelerated peer review process. 

Pursuant to Ex parte Sarallo and Ex parte Ban, the Maddison reference does not 
expressly teach and does not fairly suggest providing the completed evaluation form with author 
responses to an editor and therefore, the rejection should not be maintained. Furthermore, a 
reading of Maddison that suggests that it teaches providing the completed evaluation form with 
author responses to an editor can only be based on the benefit of hindsight gleaned from 
Appellants disclosure and therefore the rejection should not be maintained. Appellants 
respectfully request that the rejection of claim 1 be reversed by the Board. 

Mathews -providing the completed evaluation form to the author; 

The Final Action buttresses its rejection of this claim element based on Maddison with an 
additional rejection based on Mathews. The Final Action cites Mathews as teaching "providing 
the completed evaluation form to the author (Mathews, page 3, lines 33 - 35 and page 7, lines 22 

- 33; the score and the comments are mailed to the author)." 

Specifically, Mathews states: 
Peer Review Processes 

The Peer Review Processes provide a facility for managing all of 
the peer review processes of the conference, which is discussed in 
detail later. 

Mathews, page 3, Lines 36 - 38. 

Submission Reviews 

Once papers are assigned, reviewers will have access to these 
papers and be able to enter the grading information. The reviewer 
gets a list of papers, and when a paper is selected, an evaluation 
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form is filled in with the information about that paper. The 
reviewer grades the paper according to several criteria (relevance, 
originality, correctness, and quality) on a scale from 1 to 5 where 1 
is the poorest and 5 is the best score. In addition to these numeric 
scores the reviewer must also include private comments about the 
paper to understand later, for example, why relevance for a paper 
was a "4" and not a "5". These comments are important in the 
refereeing process to select borderline papers that may or may not 
be accepted. There is also a comments field to suggest 
modifications needed for the paper to be accepted, which are 
mailed to the author along with the scores. The committee may 
examine a single review or examine all reviews for a given paper 
to ensure that the scores between two reviewers do not differ by 
more than one point in which case the committee "must confer with 
the reviewers to re-evaluate their decisions and bridge the gap 
between the scores. 

The private comments and modifications to authors' comments 
were initially stored within the Oracle database assignments table, 
but problems surfaced owing to Oracle's limitations on string data 
types within SQL statements. A VARCHAR2 datatype (a variable 
character type) cannot exceed 2,000 characters, and the reviewers 
entering long reviews (> 2,000 characters) resulted in some 
comments being truncated. The WWW5 EMS stores the comments 
in external ASCII files that are outside the Oracle database but 
accessible from the EMS's forms interface as well as directly 
readable from the file system. Within the database the file name of 
the comments is stored and a special flag within the form triggers 
the CGI program to read the contents of the file and insert this into 
the form's TEXTAREA field for editing. 
Mathews, page 7, Lines 9 - 36. 

Here, Mathews is clear that the evaluation form is filled in with the information about 
that paper, including grades according to several criteria (in numerical score format), private 
comments, and modification comments. Mathews expressly states that these modification 
comments and the scores and the numerical scores are mailed to the author. What Mathews does 
not teach or suggest is that the completed evaluation form is provided to the author, as required 

by this element of claim 1 . 

Notably, the evaluation form taught by Mathews includes the private comments field and 
Mathews does not expressly teach and does not fairly suggest that the private comments are 
mailed to the author. To the contrary, Mathews teaches that the private comments field is 
necessary for later understanding of the various scores for a paper and to facilitate the refereeing 
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process for borderline papers that may or may not be accepted. Thus, Mathews does not teach 
that the completed form is provided to the author because the private comments are part of the 
completed form according to Mathews. 

Accordingly, Mathews is at best silent with respect to providing the completed evaluation 
form to the author and therefore any reading of Mathews that it teaches this step would be 
speculation and based on hindsight gleaned from reading Appellants disclosure. Ex parte 
Sarallo, at 2; Ex parte Ban, at 4. Mathews therefore fails to cure the fatal defect of Maddison 
with respect to the element of providing the completed evaluation form to the author. 
Consequently, because the cited prior art does not teach or disclose this element and the rejection 
of claim 1 should be reversed and claim 1 and its dependent claims should be allowed. 



Prima Facia Obviousness Not Met for Claim 1 

Appellants assert that the tenuous seven (7) way obviousness rejection of independent 
claim 1 fails to make a prima facia case of obviousness because no motivation to combine the 
seven (7) references has been identified in the Final Action. In particular, no expressly stated or 
implied suggestions are contained in the references to indicate to one having ordinary skill in the 
art that the seven (7) references should be combined to arrive at Appellants' method for peer 
review over a communications network. 

Specifically, the Final Action cites the Kao reference as teaching "a form is transmitted 
among several user to review includes comments which are entered directly in the form (Kao, 
col. 5, lines 8-10; col. 5, lines 17-40; and fig. 4) " Kao states: 

A form may be transmitted only from one user to another user, or it 

may be transmitted serially among several users in turn. 

Kao, Column 5, Lines 8-10. 

Associated with the Sales Forecast Report form 401 is audit trail 
410 Audit trail 410 lists the various users that receive or use the 
report. In the example of FIG. 4, the Sales Forecast Report was 
created and submitted by user James Doe. The report was then 
transmitted to user Jeff Doe for approval, and then to user John 
Doe for final approval. In one embodiment of the present 
invention, as a user receives the report, the name or identification 
of that user is appended to the audit trail. The audit trail 410 of 
Sales Forecast Report 401 contains several fields for the display of 
pertinent information regarding each user's activities regarding the 

11 
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report. Such information includes a "User" field 414 that identifies 
the user by name or similar identification (e.g., employee number), 
and an "Action" field 416 that lists the action taken on the form by 
each respective user. In the example of FIG. 4, the action field 
indicates ' that user James Doe submitted the form, and the 
subsequent recipients of the form both approved the form. The 
"Time" field 418 provides the date and time at which a user 
performed the action on the report. A comment field 420 is 
provided to allow a user to type text information pertaining to the 
report or any action taken with respect to the form, or any other 
desired information. In most implementations, each transaction 
involving a particular form or document will be assigned a 
transaction number that identifies a particular transaction. The 
"No." field 412 provides the display area for the transaction 
number associated with each audit trail entry. These numbers may 
be assigned chronologically so that higher numbers correspond to 
later transactions. 
KaOy Column 6 y Lines 17-45. 

Fig. 4 is reproduced as Exhibit E. 

Kao teaches an audit history for a database form. The audit history contains information 
describing the transmission history of the document and any action taken with regard to the 
document. The audit history is updated automatically as the document is transmitted from one 
user to another user in the network. (Kao, Column 2, Lines 1 0 - 23). Kao teaches that a form 
can be transmitted between users in a network (Kao, Column 5, Lines 8-10) and provides as an 
example a description of a Sales Forecast Report (Fig. 4) that is passed to three users. (Kao, 
Column 6, Lines 17 - 45). The audit trail of the three users is shown as element 410 in Fig. 4. 

What Kao fails to disclose is any reason, suggestion, motivation or teaching to combine 

its database form audit history to peer review of articles. The Final Action states that "[i]t would 

have been obvious to a person of ordinary skill in the art at the time the invention was made to 

have modified Kao into Maddison, Pope, and Mathews to allow the author responses [sic] to the 

reviewers evaluation form directly in the completed evaluation form and providing the 

completed evaluation with author responses to an editor to facilitate the peer review process, 

since this would have helped the reviewers, authors, and editor are [sic] able to use the 

evaluation form to communicate to each other in peer review conducted on-line." The Final 

Action, however, fails to identify a motivation, teaching or suggestion to combine the prior art to 

provide a method for peer review as claimed by Appellants. Thus, the Final Action fails to make 
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a prima facia case of obviousness because no teaching, suggestion, or motivation to combine is 
found in the references. 

Claim 6 stands alone. 

Dependent claim 6 is separately patentable because it introduces the concepts of 
reformatting the article into a standard format and then presenting the reformatted article to the 
author for approval. The advantages of a standard format and a quality review by the author 
prior to sending the article to a reviewer are numerous and specifically include the insurance that 
the article that is ultimately provided to a reviewer and ultimately an editor is presented as the 
author intends and in the format that the reviewer or editor expects. Reformatting of a document 
into a standard format is not trivial and certainly many formatting errors can occur such as the 
decoupling of text and figures, just to name one example. Furthermore, allowing an author to 
approve the reformatted article at this point in the process facilitates the initial review and makes 
the entire review process more efficient. Thus, reformatting the document into a standard format 
and presenting the reformatted article to the author for approval are discrete concepts that are 
separately patentable from the claimed computer implemented method for peer review presented 
in the base claim. 

The Final Action cites the Pope reference as teaching the step of presenting the final 
version article to the author for approval (Pope, page 7, lines 7-9; see also page 5, under the 
heading "Manuscript tracking and peer review: publication" in Appellants' version). The Final 
Action acknowledges that Pope does not expressly teach the claimed element of presenting the 
reformatted article to the author for approval. The Final Action states that it would have been 
obvious to present the reformatted article to the author for approval because the final version of 
the article is presented to the author. 

Here, as in In re Ban, the examiner's reading of what is taught or suggested in Pope 
appears to be based on hindsight gleaned from the benefit of reading Appellants' disclosure 
rather than on the fair teachings of the reference. Therefore, Appellants respectfully request that 
the rejection of claim 6 be reversed. 
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B. Whether claims 9, 10, 1 1, 12, 15, 19, 20, 23, and 24 are patentable under 35 
U.S.C. 103 over the combination of (1) Pope; (2) Maddison; (3) Mathews; (4) 
Kao; (5) Sumner; (6) Walker; and (7) Borovoy. 

Claims 9. 10, 15. 19. 20/23 ■ and 24 in G roup B above stand or fall together. 

Independent claim 9 was rejected in the Final Action based on a combination of the 
references listed above. In particular, the rejection relies on Maddison and Mathews which are 
collectively cited as teaching certain steps as follows: 

Maddison -providing the author with the completed evaluation form; 

-receiving a response from the author directly in the completed evaluation form; 
-providing the completed evaluation form with author responses to an editor; 

Mathews -providing the author with the completed evaluation form; 

These references, however, standing alone or in combination fail to teach the three 
elements as demonstrated below. 

Maddison - providing the author with the completed evaluation form; 

The Final Action cites Maddison at page 2, lines 27 - 29 and 33 as teaching this element. 

Specifically, Maddison states: 

Reviews are usually conducted on-line, with the reviewer typing 
comments about individual sections of a contribution into the 
windows of a structured evaluation form provided on a special 
peer-review web page. 
Maddison, Page 2, Lines 36 - 38. 

The comments furnished by the reviewers are passed along to the 
authors. 

Maddison, Page 3, Line 1 . 

Upon review of the cited passages, Maddison expressly teaches that the reviewer types 
comments into a structured evaluation form that is provided on a special peer-review web page. 
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It is also expressly taught that those comments are passed along to the authors. What Maddison 
fails to teach is that the completed evaluation form is provided to the author, as required by claim 
9. At best, Maddison is silent with respect to teaching that the completed evaluation form is 
provided to the author because its express teachings only state that the comments are passed 
along to the author. Importantly, Maddison does not expressly teach that the completed 
evaluation form is provided to the author. 

As previously discussed, in Ex parte Sarallo, the Board found that there was no express 
teaching of the subject matter asserted by the examiner to be present at the cited location in the 
reference. Ex part Sarallo, at 2. Furthermore, the Board found that the reference relied upon by 
the Examiner was silent as to the operation of the claimed element. Id. Maddison is equally 
silent as to providing the completed evaluation form to the author. In Ex parte Sarallo, the 
Board reversed the 103 rejection of the claim and stated that to interpret the cited reference as 
teaching the claimed element would be speculation on its part, which the Board stated it would 
not do. Id. Appellants request that the Board also refrain from speculation here and reverse the 
rejection of claim 9. 

As also previously discussed, in Ex parte Ban, the Board found that each specification of 
two references were silent as to the spatial relationship of certain elements required by the claim, 
even though the figures in both references showed the elements arranged with spacing. Ex parte 
Ban, at 3. The Board stated that under the circumstances, where the references were silent, the 
examiner's position that the reference suggested the claimed element was not well taken. Id. , at 
4. Moreover, the Board further acknowledged that the examiner's reading of what the references 
suggested was based on hindsight gleaned from reading the Appellants disclosure rather than on 
the fair teachings of the references. Similarly here, a reading of Maddison that it suggests 
providing the completed evaluation form to the author necessarily requires the benefit of 
Appellants' disclosure because such a step is not expressly taught or fairly suggested by 
Maddison. Accordingly, Appellants respectfully request that the rejection of claim 9 be 
reversed. 
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Maddison -receiving a response from the author directly in the completed evaluation form; 

The Final Action cites Maddison as teaching "receiving a response from the author 
(Maddison, page 3, lines 1 - 7, teaches that the author responses [sic] to the reviewers when the 
author decline [sic] to make the requested changes from the reviewer)." Specifically, Maddison 
states: 

The comments furnished by the reviewers are passed along to the 
authors. If the reviewers recommend that a contribution be 
revised, the authors are asked to amend the page and resubmit it to 
the editor. Authors may at this point decline to make the requested 
changed, but they would be expected to support their opinion with 
plausible arguments. Any such explanation -would first be 
considered by the editor-in-chief in consultation with associate 
editors, but it might also be forwarded to the reviewers for further 
comment. 

Maddison, Page 3, Lines 1-7. 

The cited passage here states that an author is expected to support her decision not to 
amend and resubmit the page with plausible arguments. Maddison further states that any such 
arguments would be considered by the editors and possibly the reviewers. Thus, although 
Maddison teaches that responses from the authors are received, what Maddison fails to teach is 
that the responses from the author are received directly in the completed evaluation form, as 
required by claim 9. At best, Maddison is silent with respect to teaching that the author's 
comments are received directly in the evaluation form because its express teachings only state 
that "authors would be expected to support their opinion with plausible arguments" and that 
"[a]ny such explanation would first be considered by the editor-in-chief. . .". Importantly, 
Maddison does not expressly teach receiving a response from the author directly in the 
completed evaluation form. 

Ex parte Sarallo and Ex parte Ban are equally applicable here because this claim element 
is not expressly taught or fairly suggested by Maddison. Thus, any reading of Maddison by 
Board that it teaches or suggests receiving a response from the author directly in the completed 
evaluation form would be speculation. Moreover, similar to Ex parte Ban, the position in the 
Final Action that Maddison, although silent, teaches this step suggests that the examiner's 
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reading of Maddison is based on hindsight gleaned from reading the Appellants disclosure rather 
than on the fair teachings of the reference. Accordingly, Appellants respectfully request that that 
the Board refrain from speculation and hindsight here and that the rejection of claim 9 based on 
Maddison be reversed. 

Maddison -providing the completed evaluation form with author responses to an editor; 

The Final Action cites Maddison as teaching "providing the author responses to an editor 
(Maddison, page 3, lines 1 - 7, teaches that the editors consider the author response.)" The 
language from Maddison, page 3, lines 1 - 7 is set forth in full above. 

Notably, the claim element is not merely the limitation that the author's responses are 
provided to an editor as the examiner suggests. Rather, the claimed element is providing the 
completed evaluation form with author responses to an editor. The claimed limitation is 
considerably different than the limitation rejected by the examiner and the cited passage does not 
fairly teach or suggest the claimed element. 

As with the elements addressed above, Maddison is at best silent with respect to teaching 
the claimed element of providing the completed evaluation form with author responses to an 
editor. As explained in the specification (page 18, line 4 - page 19, line 4; and page 19, lines 14 
- 16), this point by counter-point manner of presenting the reviewer's comments and the author's 
responses simplifies the editor's examination of the article, the review and the corresponding 
remarks, which in turn expedites the publication recommendation and decision and thereby 
reduces the time to publication for the article. Notably, a clear advantage associated with 
providing the completed evaluation form with author responses to an editor includes an 
accelerated peer review process. 

Pursuant to Ex parte Sarallo and Ex parte Ban, the Maddison reference does not 
expressly teach and does not fairly suggest providing the completed evaluation form with author 
responses to an editor and therefore, the rejection should not be maintained. Furthermore, a 
reading of Maddison that suggests that it teaches providing the completed evaluation form with 
author responses to an editor can only be based on the benefit of hindsight gleaned from 
Appellants disclosure and therefore the rejection should not be maintained. Appellants 
respectfully request that the rejection of claim 9 be reversed by the Board. 
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Mathews -providing the completed evaluation form to the author; 

The Final Action 'buttresses its rejection of this claim element based on Maddison with an 
additional rejection based on Mathews. The Final Action cites Mathews as teaching "providing 
the completed evaluation form to the author (Mathews, page 3, lines 33 - 35 and page 7, lines 22 
- 33; the score and the comments are mailed to the author) 

Specifically, Mathews states: 
Peer Review Processes 

The Peer Review Processes provide a facility for managing all of 
the peer review processes of the conference, which is discussed in 
detail later. 

Mathews, page 5, Lines 36 - 38. 

Submission Reviews 

Once papers are assigned, reviewers will have access to these 
papers and be able to enter the grading information. The reviewer 
gets a list of papers, and when a paper is selected, an evaluation 
form is filled in with the information about that paper. The 
reviewer grades the paper according to several criteria (relevance, 
originality, correctness, and quality) on a scale from 1 to 5 where. 1 
is the poorest and 5 is the best score. In addition to these numeric 
scores the reviewer must also include private comments about the 
paper to understand later, for example, why relevance for a paper 
was a "4" and not a "5". These comments are important in the 
refereeing process to select borderline papers that may or may not 
be accepted. There is also a comments field to suggest 
modifications needed for the paper to be accepted, which are 
mailed to the author along with the scores. The committee may 
examine a single review or examine all reviews for a given paper 
to ensure that the scores between two reviewers do not differ by 
more than one point in which case the committee must confer with 
the reviewers to re-evaluate their decisions and bridge the gap 
between the scores. 

The private comments and modifications to authors' comments 
were initially stored within the Oracle database assignments table, 
but problems surfaced owing to Oracle's limitations on string data 
types within SQL statements. A VARCHAR2 datatype (a variable 
character type) cannot exceed 2,000 characters, and the reviewers 
entering long reviews (> 2,000 characters) resulted in some 
comments being truncated. The WWW5 EMS stores the comments 
in external ASCII files that are outside the Oracle database but 
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accessible from the EMS's forms interface as well as directly 
readable from the file system. Within the database the file name of 
the comments is stored and a special flag within the form triggers 
the CGI program to read the contents of the file and insert this into 
the form's TEXTAREA field for editing. 
Mathews, page 7, Lines 9-36. 

Here, Mathews is clear that the evaluation form is filled in with the information about 
that paper, including grades according to several criteria (in numerical score format), private 
comments, and modification comments. Mathews expressly states that these modification 
comments and the scores and the numerical scores are mailed to the author. What Mathews does 
not teach or suggest is that the completed evaluation form is provided to the author, as required 

by this element of claim 9. 

Notably, the evaluation form taught by Mathews includes the private comments field and 
Mathews does not expressly teach and does not fairly suggest that the private comments are 
mailed to the author. To the contrary, Mathews teaches that the private comments field is 
necessary for later understanding of the various scores for a paper and to facilitate the refereeing 
process for borderline papers that may or may not be accepted. Thus, Mathews does not teach 
that the completed form is provided to the author because the private comments are part of the 
completed form according to Mathews. 

Accordingly, Mathews is at best silent with respect to providing the completed evaluation 
form to the author and therefore any reading of Mathews that it teaches this step would be 
speculation and based on hindsight gleaned from reading Appellants disclosure. Ex parte 
Sarallo, at 2; Ex parte Ban, at 4. Mathews therefore fails to cure the fatal defect of Maddison 
with respect to the element of providing the completed evaluation form to the author. 
Consequently, because the cited prior art does not teach or disclose this element and the rejection 
of claim 9 should be reversed and claim 9 and its dependent claims should be allowed. 

Prima Facia Obviousness Not Met for Claim 9 

Appellants assert that the tenuous seven (7) way obviousness rejection of independent 
claim 9 fails to make a prima facia case of obviousness because no motivation to combine the 
seven (7) references has been identified in the Final Action. In particular, no expressly stated or 
implied suggestions are contained in the references to indicate to one having ordinary skill in the 
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art that the seven (7) references should be combined to arrive at Appellants' method for peer 
review over a communications network. 

Specifically, the Final Action cites the Kao reference as teaching "a form is transmitted 
among several user to review includes comments which are entered directly in the form (Kao, 
col. 5, lines 8 - 10; coL 5, lines 17-40; and fig. 4)." Kao states: 

A form may be transmitted only from one user to another user, or it 
may be transmitted serially among several users in turn. 
Kao, Column 5, Lines 8 - 10. 

Associated with the Sales Forecast Report form 401 is audit trail 
410. Audit trail 410 lists the various users that receive or use the 
report. In the example of FIG. 4, the Sales Forecast Report was 
created and submitted by user James Doe. The report was then 
transmitted to user Jeff Doe for approval, and then to user John 
Doe for final approval. In one embodiment of the present 
invention, as a user receives the report, the name or identification 
of that user is appended to the audit trail. The audit trail 410 of 
Sales Forecast Report 401 contains several fields for the display of 
pertinent information regarding each user's activities regarding the 
report. Such information includes a "User" field 414 that identifies 
the user by name or similar identification (e.g., employee number), 
and an "Action" field 416 that lists the action taken on the form by 
each respective user. In the example of FIG. 4, the action field 
indicates that user James Doe submitted the form, and the 
subsequent recipients of the form both approved the form. The 
"Time" field 418 provides the date and time at which a user 
performed the action on the report. A comment field 420 is 
provided to allow a user to type text information pertaining to the 
report or any action taken with respect to the form, or any other 
desired information. In most implementations, each transaction 
involving a particular form or document will be assigned a 
transaction number that identifies a particular transaction. The 
"No." field 412 provides the display area for the transaction 
number associated with each audit trail entry. These numbers may 
be assigned chronologically so that higher numbers correspond to 
later transactions. 
Kao, Column 6, Lines 17-45. 

Fig. 4 is reproduced as Exhibit E. 

Kao teaches an audit history for a database form. The audit history contains information 
describing the transmission history of the document and any action taken with regard to the 
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document. The audit history is updated automatically as the document is transmitted from one 
user to another user in the network. (Kao, Column 2, Lines 10 - 23). Kao teaches that a form 
can be transmitted between users in a network (Kao, Column 5, Lines 8 - 10) and provides as an 
example a description of a Sales Forecast Report (Fig. 4) that is passed to three users. (Kao, 
Column 6, Lines 17 - 45). The audit trail of the three users is shown as element 410 in Fig. 4. 

What Kao fails to disclose is any reason, suggestion, motivation or teaching to combine 
its database form audit history to peer review of articles. The Final Action states that "[i]t would 
have been obvious to a person of ordinary skill in the art at the time the invention was made to 
have modified Kao into Maddison, Pope, and Mathews to allow the author responses [sic] to the 
reviewers evaluation form directly in the completed evaluation form and providing the 
completed evaluation with author responses to an editor to facilitate the peer review process, 
since this would have helped the reviewers, authors, and editor are [sic] able to use the 
evaluation form to communicate to each other in peer review conducted on-line." The Final 
Action, however, fails to identify a motivation, teaching or suggestion to combine the prior art to 
provide a method for peer review as claimed by Appellants. Thus, the Final Action fails to make 
a prima facia case of obviousness because no teaching, suggestion, or motivation to combine is 
found in the references. Appellants therefore respectfully request that the rejection of claim 9 
and its dependent claims be reversed by the Board. 

Claims 1 1 and 12 stand or fall together. 

Claim 1 1 is patentably distinct because it introduces the concept of allowing a reviewer to 
gain access to reviews of the same article that were completed by other co-reviewers. The 
advantages of doing so are numerous and include raising the experience level of a reviewer by 
exposure to reviews of the same article and homogenizing the format and style of future reviews 
so that the review process can be more efficient. Thus, allowing reviewers to self-critique their 
own work by contemplating other reviewers' work is a discrete concept that is separately 
patentable from the claimed computer implemented method for peer review presented in the base 
claim. 

In its rejection, the Final Action admits that Mathews does not disclose a reviewer 
gaining access to other completed evaluation forms after submitting an evaluation form for the 
same article. Yet, the Final Action maintains that it would have been obvious to do so "to 
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provide honesty [sic] evaluation, since examining other completed evaluation form for the same 
article before evaluate and submit it would have influenced the reviewer' point of view in 
evaluating process." 

Similar to In re iBan, the examiner's reading of what is taught or suggested in Mathews 
appears to be based on hindsight gleaned from the benefit of reading Appellants' disclosure 
rather than on the fair teachings of the reference. The Final Action provides no disclosure, 
express or suggested, that states or implies that reviewers gain access to completed evaluation 
forms after submitting a completed evaluation form for the same article. The prior art does not 
teach this step and therefore, Appellants respectfully request that the rejection of claim 1 1 and its 
dependent claim 12 be reversed. 

C. Whether claim 4 is patentable under 35 U.S.C. 103 over the combination of 
(1) Maddison; (2) Pope; (3) Mathews; (4) Kao; (5) Sumner; (6) Walker; (7) 
Borovoy; and (8) Sato. 

Claim 4 requires that upon receipt of an article from an author, a ranked list of keywords 
related to the article is generated, the ranked list of keywords is provided to the author, and the 
author approves the ranked list of keywords. This concept is advantageous because it allows the 
author a measure of quality control over the computer implemented process for deriving 
keywords from the article, and indirectly allows a measure of quality control of the assignment 
of reviewers for the article. 

The Final Action cites the Pope reference as teaching the step of receiving an approval of 
the final version article from the author (Pope, page 7, lines 7-9; see also page 5, under the 
heading "Manuscript tracking and peer review: publication" in Appellants' version). The Final 
Action, however, after acknowledging that Pope does not expressly teach the claimed steps for 
generating a ranked list of keywords, providing the list to the author, and receiving an approval 
of the list from the author, goes on to state that the claimed steps are obvious because presenting 
the ranked list of keywords to the author is the same as presenting the final version of the article 
to the author (as taught by Pope). 

The Final Action fails to show how these two separate ideas (and separate claims) are the 

same. The Final Action further fails to show any suggestion derived from the references that 
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teach or disclose the claimed steps. The Final Action further fails to demonstrate any motivation 
to modify Pope's teaching of author approval for the final version of the article to provide for 
author approval of a ranked list of keywords. Thus, it seems that as in the In re Ban case, the 
examiner's reading of what is taught or suggested in Pope appears to be based on hindsight 
gleaned from the benefit of reading Appellants' disclosure rather than on the fair teachings of the 
reference. Therefore, Appellants respectfully request that the rejection of claim 4 be reversed. 

D. Whether claim 16 is patentable under 35 U.S.C. 103 over the combination of 
(1) Pope; (2) Maddison; (3) Mathews; (4) Kao; (5) Sumner; (6) Walker; (7) 
Borovoy; and (8) Sato. 

Claim 16 further limits the searching the database step of independent claim 9 and 
requires that the title and text of the article are parsed to generate a list of keywords, that the list 
of keywords is ranked according to their relative weight in describing the content of the article, 
and that the database is searched to generate a list of qualified reviewers. 

The Final Action cites Sumner as implying "the step of parsing the article, such as the 
title, abstract, and text content to find [the] article's keywords for matching the reviewer 
records." Specifically, Sumner states: 

Currently, we are witnessing the beginnings of a shift from paper 
to digital media in the submission, review and publication of 
scholarly work. In many journals, it is now standard procedure for 
reviews and submissions to be transmitted electronically. 
Sometimes, automated systems are in place for matching reviewers 
with submissions based on keyword analysis. In a few cases, the 
final version is even published electronically as a set of HTML 
files available on the WWW. 
Sumner, Page 2, Lines 4-9. 

A review of the passage shows that Sumner expressly teaches that automated systems are 
in place for matching reviewers with submissions based on keyword analysis. What Sumner 
does not expressly teach or disclose is how the keywords are generated. Sumner does not teach 
or suggest that the keywords are generated by parsing the title and text of the article, as required 
by claim 1 6. At best, Sumner is silent as to parsing the title and text of the article to generate a 
list of keywords. 
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Ex parte Sarallo and Ex parte Ban are applicable here because this claim element is not 
expressly taught or fairly suggested by Sumner. The reference is at best silent. Accordingly, any 
reading of Sumner by Board that it teaches or suggests parsing the title and text of the article to 
generate a list of keywords would be speculation, which the Board stated in Ex parte Sarallo that 
it would not do. Moreover, similar to Ex parte Ban, the position in the Final Action that Sumner, 
although silent, teaches this step suggests that the examiner's reading of Sumner is based on 
hindsight gleaned from reading the Appellants disclosure rather than on the fair teachings of the 
reference. Accordingly, Appellants respectfully request that that the Board refrain from 
speculation and hindsight here and that the rejection of claim 16 be reversed. 

Conclusion 

Appellants assert that the tenuous seven (7) way obviousness rejection of independent 
claims 1 and 9 fail to make a prima facia case of obviousness because no motivation to combine 
the references has been identified in the Final Action. Additionally, the rejections of the claimed 
elements as set forth in the Final Action fail to teach the elements that are claimed by Appellants. 
In particular, the references are silent as to the claimed elements and would require speculation 
to be read on Appellants' claimed steps. Finally, the readings of the references as set forth in the 
reasoning for the rejections in the Final Action seem to be impermissibly based on hindsight 
gleaned from the examiner's knowledge and reading of Appellants disclosure rather than on the 
fair teachings of the references. Appellants therefore assert that all pending claims are currently 
in condition for allowance and that the rejections set forth in the Final Action should be reversed 
by the Board. 

Respectfully submitted, 

Procopio, Cory, Hargreaves & Savitch llp 



Dated: Se ptember 18.2003 



By 




Pattric J. (Rawlins 
Reg. No. 47,887 



Procopio, Cory, Hargreaves & Savitch LLP 
530 B Street, Suite 2100 
San Diego, California 92101-4469 
(619) 238-1900 
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(9) Appendix 

1 . (Twice Amended) A computer implemented method for peer review over a communications 
network, comprising: 

receiving an article from an author via a communications network; 
extracting context information from the article; 

querying a database containing profiles of potential reviewers to determine a qualified 
reviewer based on the extracted context information; 
assigning the qualified reviewer to the article; 
providing an evaluation form to the reviewer; 
receiving a completed evaluation form from the reviewer; 
providing the completed evaluation form to the author; 

receiving a response from the author directly in the completed evaluation form; 
providing the completed evaluation form with author responses to an editor; 
receiving a publication decision from the editor; and 
providing the publication decision to the author and the reviewer. 

2. Canceled. 

3. (Amended) The method of claim 1 whereby the qualified reviewer is determined according 
to availability and past performance, including timeliness, thoroughness, clarity, and the 
number of completed reviews. 

4. (Amended) The method of claim 3 whereby the receiving an article step further comprises: 

generating a ranked list of keywords related to the article; 
providing the list of keywords to the author; and 
receiving an approval of the ranked list from the author. 

5. (Amended) The method of claim 1 whereby the receiving a completed evaluation form step 
further comprises: 

receiving comments entered directly into the evaluation form. 
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6. (Amended) The method of claim 1 whereby the receiving an article step further comprises: 

automatically reformatting the article into a standard format; and 
presenting the reformatted article to the author for approval. 

7. Canceled. 

8. (Amended) The method of claim 1 whereby the providing the publication decision step 
further comprises: 

informing the author of the publication decision by email. 

9. (Twice Amended) A computer implemented method for peer review over a communications 
network, comprising: 

receiving an article from an author via a communications network; 
extracting context information from the article; 

querying a database containing profiles of potential reviewers to determine a plurality of 
qualified reviewers based on the extracted context information; 
ranking the plurality of qualified reviewers; 

contacting each qualified reviewer and requesting that the qualified reviewer agree to 
review the article; 

receiving an agreement from one or more qualified reviewers; 

providing the article to an accepting qualified reviewer; 

providing an evaluation form to the accepting qualified reviewer; 

receiving a completed evaluation form from the accepting qualified reviewer; 

providing the author with the completed evaluation form; 

receiving a response from the author directly in the completed evaluation form; 

providing the completed evaluation form with author responses to an editor; 

receiving a publication decision from the editor; and 

providing the publication decision to the author and the reviewer. 

10. (Amended) The method of claim 9 whereby the accepting qualified reviewers complete the 
evaluation form online. 
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1 1 . (Amended) The method of claim 10 whereby each accepting qualified reviewer for an article 
has access to completed evaluation forms of other accepting qualified reviewers after 
submitting an evaluation form for the same article. 

12. (Amended) The method of claim 1 1 whereby each accepting qualified reviewer can set 
access privileges for a section of the reviewer's completed evaluation form. 

13. Canceled. 

14. Canceled. 

15. (Amended) The method of claim 9 whereby the receiving an article step further comprises: 

reformatting the article into a standard format. 

16. (Amended) The method of claim 9 whereby the searching a database step further comprises: 

parsing the title and text of the article to generate a list of keywords; 
ranking the list of keywords according to their relative weight in describing the content of 
the article; and 

searching the database to generate a list of qualified reviewers. 

17. Canceled. 

18. Canceled. 

1 9. (Amended) The method of claim 9 whereby the article iterates through the peer review 
process until the article is approved for publication. 

20. (Amended) The method of claim 19 whereby after the article has been approved for 
publication, further comprising: 

creating a galley proof of the article; 

providing the galley proof to the author and editor; 

receiving an approval of the galley proof from the author and editor; and 

immediately publishing the article in an electronic format. 

21. Canceled. 
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22. Canceled. 

23. (Amended) The method of claim 9 whereby a group of authors collaborate to co-author an 
article, further comprising: 

receiving a co-authored article via a communications network. 

24 . (Amended) The method of claim 23 , further comprising: 

receiving a response from each co-author directly in the completed evaluation form. 
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EXHIBIT A 



Peer Review 



Peer-review, the critical evaluation of manuscripts by professional colleagues, is 
5 the traditional method of quality control in science. In order to improve the 
overall quality of contributions and to increase the involvement of the scientific 
community in the continuing development of the project, the Tree of Life now 
implements a formal peer-review system. Currently, there are only a few peer- 
reviewed pages on the Tree, but their number is steadijy increasing. 
10 On this page, you will find information on the following topics: 
. Peer Review and the Structure of the Tree of Life Project 

• The Peer-Review Process 



15 Peer Review and the Structure of the Tree of Life Project 

Contributions to the Tree of Life are usually by invitation from a coordinator. 
Coordinators are themselves usually authors of the basal pages for a given 
group, i.e., each author of a Tree of Life page serves as the coordinator for the 
pages descending from his or her clade: 

Accessory Genera Species 




Aut hor : Jane Doe Coordi nat or : .Jane Doe 



Coordinators help choose people to author and coordinate subgroups, and they 
serve as Associate Editors for this particular section of the Tree. Thus, the 
project currently functions more as a multi-chapter book, with authors chosen by 
the editors and associate editors, and less as a scientific journal, with submitted 
25 papers. 



I 



Paaes on the Tree vary in their degree of completeness, from skeletal pages 
L^i g as links between complete pages, to complete pages that have been 
peer-reviewed. More discussion of levels of completeness can be found on the 
paqe finalit y Assurance . 

5 By soliciting the critical input of independent peer-reviewers, we hope to move 
closer to our goal of ensuring that each Tree of Life page provides a 
comprehensive, balanced synthesis of the current views about a given group of 
organisms, including the phylogeny of the group. We plan to have each 
contribution to the Tree of Life will go through peer-review even ually but 

i o oreliminary drafts of pages are usually connected to the Tree before the 
dSnTt ve peer-reviewed version is ready to be published. Since the review 
process is labor-intensive and time-consuming, there may be considerable delays 
• beS tSfl^ appearance of a page on the Tree and its final acceptance as a 
previewed contribution. Should the page be rejected, it may remain on the 

15 Tree as a temporary page, marked with the "Under Construction. 



. Only those pages that are labelled with a peer-review icon Ei-Jhave 
successfully completed the review process. 

Once a page has successfully completed the peer-review proces i, thai and all 
subsequent versions will be archived, and access will be provided to these 
20 previous versions. 



The Peer- Review Process 

The Tree of Life peer-review works similar to the procedure adopted by many 

25 scientific journals. Pages that are to undergo peer-review are submitted to two 
to three independent subject area specialists. Reviewers are chosen by the 
editor-in-chief (David Maddison) in consultation with the associate ed. tors (i.e., 
coordinators ^) of a given group. The major criterion for the selection of potential 
reviewers is their scholarly expertise as demonstrated by their publication record 

30 on the organisms represented on the page to be reviewed. The reviews are 
administered under the system of one-sided anonymity, i.e reviewers ; are 
provided with the names of the authors, but authors are not told the idenbt.es ; of 
the reviewers; reviewers do, however, have the option of revealing their identity 
to the authors, if they wish to do so. 

35 Reviews are usually conducted on-line, with the reviewer typing comments about 
individual sections of a contribution into the windows of a structured evaluation 
form provided on a special peer-review web page. An explicit definition of the 
review criteria is given on a linked page, Notes for Reviewers, which summarizes 
our guidelines for the design and content of contributions to the Tree of Life. 

40 Reviewers are expected base their recommendation as to a page s acceptance or 
rejection on a thorough examination of the submitted material. If a page is 
deemed acceptable, reviewers have the option of requesting either major or 
minor revisions that would improve the page's content. 



The comments furnished by the reviewers are passed along to the authors. If the 
reviewers recommend that a contribution be revised, the authors are asked to 
amend the page and resubmit it to the editor. Authors may at this point decline 
to make the requested changes, but they would be expected to support their 
5 opinion with plausible arguments. Any such explanation would first be considered 
by the editor-in-chief in consultation with associate editors, but it might also be 
forwarded to the reviewers for further comment. In the event of contradictory 
evaluations by reviewers or of disagreement between reviewers and authors, 
additional re viewers may be engaged. A Tree of Life page is awarded the peer- 

10 review mark 



,^ Y1 _. once authors have made all the changes and additions 

required by the editor on the basis of the reviewers' recommendations. If the 
peer-review process results in a rejection of a contribution, the version of the 
page on the Tree of Life will return to a preliminary status, and we will initiate 
search for new authors for this page. 
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Abstract 

This paper deals with managing the peer review process with an electronic 
management system (EMS). Our approach uses a WWW-based interface to an 
Oracle database. We describe different aspects of the EMS, namely the peer 
review process, report generation, and security. An experimental peer review 
system was developed for the Fourth International World Wide Web (WWW) 
Conference, which provided a model to automate the underlying processes that 
had been driven primarily by pen and paper. This system uses the very technology 
that the International WWW Conference Committee (IW3C2) promotes, so it is 
fitting that the conference be automated with a WWW-based system. The 
experimental system has evolved with the help of many suggestions to provide 
better support for the WWW5 conference as well. This paper disciisses the first 
experimental system, some lessons learned, and the second more evolved system 
for the WWW5 conference with emphasis on the latter. 
Keywords: 

Peer review, Electronic management system, CGI, Oracle, database, SQL 

Contents 

• Introduction 

• The Electronic Management System 

• Peer Review Processes 

o Initiation 

o Submission Review 

o Rankings and Selections 

o Announcements and Debrie fines 

• CGI and the Underlying Database 



o Searching the Database 



• Security and Authentication 



. Report Tools for Analyzing the Data 
o Standard Report Tools 
o Ad Hoc Report Tools 
5 • Conclusion 

• Acknowledgments 

• References 

. Appendix. Results of the VVWW4 Peer Review Process 

• About the Authors 

10 Introduction 

This paper addresses the electronic peer review process problem. Namely, how 
does one electronically manage the complex process of peer reviewing papers 
over a physically distributed set of participants (i.e., authors, reviewers, and 
administrators). The peer review of papers submitted to the International World 
15 Wide Web (WWW) Conference Committee (IW3C2) would serve as an example 
of this process. 

The peer review process (also called refereeing) can be thought of as a logistical 
problem that is relatively simple in concept. The peer review committee gets 
submissions, categorizes them, sends them to reviewers, collects reviews, makes 
20 final selections, and notifies submitters of the outcome. The process, however, 
gets complex in the implementation because of the scale involved and the 
geographic diversity of the program participants. There are several related 
activities in the literature [1, 2, 3]. 

Our solution to electronically managing the peer review process applies the 
25 methodology called Electronic Management Systems (EMS). In particular, we 
represent the entire peer review process as a virtual organization that uses the 
Web as a communications vehicle. The menus of the Web pages represent 
electronic processes with examples such as peer review processes and user 
management processes. Key components of EMSs are forms, report tools, and 
30 databases. 

Many EMSs have been implemented by the National Aeronautics and Space 
Administration (NASA) to provide a "paperless" capability [4] that facilitates the 
operation of complex processes, such as the peer review process. Two such 
systems handled the peer review processes of the Fourth and Fifth International 
35 WWW Conferences (WWW4 [5] and WWW5 [6]) that accepted papers worldwide 
and allowed the conference committee members and reviewers to examine and 
evaluate them from a WWW forms-based interface to an Oracle Relational 



Database Management System (RDBMS). There are also related research 
projects that have developed WWW interfaces to Oracle [2, fi] , 
ThP annrnach taken in developing these systems extends the electronic peer 
rSalih* sev'era 9 ! ways. Fr* the fi^^^^ 

5 system requires little or no programming. Second, this .s the f rst _to ally Web 
based approach for modeling the complete peer review Th. ^ we 

represent all the subprocesses of the processes in terms of HTML forms and 
Zo7£o\s on databases. Fourth, we provide a common set of report tools 
dynamically generated from database schema that will work for all databases, 

10 hence, processes. 

The Electronic Management System 

ThP rnnference oeer review EMSs (WWW4 EMS and WWW5 EMS) were designed 
for three 1) authors who submit their paper to the conference; 2) 
n^SSS slmfevaLtions of papers assigned to them and ^erence 
1 5 committee members who assign papers to reviewers and make the decision to 

^7^Twi*nTJwN5 EMS are similarly designed, however, the ~ 
EMS reorganized the processes in a more intuitive layout. Throughout this paper 
the term "EMS" will generally refer to both systems except when indicated 

20 The^MS** a collection of hyperlinked menus, Under 
traverse a hierarchy of functions belonging to steps that define a process. Unde r 
h s top layer is a relational database of four tables ( sub ^ l0 j^ 3 T f ' 9 ^f^Gn 
users and suggestions), and an assortment of Common Gateway Interface (CGI) 

25 iripts and auxiliary programs that access the database. The top-level menu 
Smrture of the EMS provides links to the following processes of the system: 

An o^rview HTML document is provided at each level of the menu hierarchy to 
give an overview of the available process steps and options. 

30 f^elulletin Board provides quick access and communication of Nrjtom^Ion. For 
example, from the Bulletin Board page there is a menu . em that ^ to the 
Reviewers Bulletin Board and another to the Administrator Bui 
purpose of the Reviewers Bulletin Board is to provide a user mterface to reviewer 

35 information and on-line forms to enter evaluations. The purpose of the 
Administrator Bulletin Board is to provide a user interface to forms and 
information about the entire system, which is edited by the administrators for a 
hotlist of links into various parts of the system. 

Peer Review Processes reuiew 
40 The Peer Review Processes provide a facility for managing all of the peer review 
processes of the conference, which is discussed in detail later. 
Users/Internal Staff Processes 



The users menu allows the administrator to create new users and update/delete 
existing users. The users include administrators (EMS staff and conference 
committee members) and reviewers. Associated with every user is a user name 
and password through which the httpd server authenticates the user for various 

5 functions of the system. Thus, reviewers can enter only reviews of papers 
assigned to them; the administrator can access all the data; and people without 
authorization (no user name or password) can use only the public access forms 
to submit a paper. 
Suggestions Processes 

10 The Suggestions Processes provide a facility for processing and tracking 

suggestions for improvement from its users. The menu provides an on-line form 
for users to enter suggestions and comments about the EMS, and another on- 
line form for administrators to respond to the suggestions. 
Report Tools 

15 The Report Tools provide a facility for making available. tools for generating 
reports across all the databases corresponding to all the above processes. Each 
level of the EMS provides a collection of report tools that report on the 
associated database with that level. For example, the report tools submenu at 
the Users/Internal Staff Processes menu provides a report of all users of the 

20 system (including user ID, full name, and E-mail address), which is an embedded 
Structured Query Language (SQL) statement as input to a CGI program that 
queries the Oracle database and formats the output in HTML 

Peer Review Processes 

In this section we discuss the peer review processes, which represent the largest 
25 and most important part of the system. These processes involve soliciting 
potential authors for submissions and managing the incoming submissions as 
well as the evaluations of them. These aspects are classified as Initiation, 
Review, Rankings and Selections, Announcements and Debriefings where each 
process is further defined with the following subprocesses: 

30 1. Initiation 

Call for Papers 
Intent to Submit 
Submissions Upload 

2. Submission Review 

35 Examination of Submissions 

Assignment of Submission Responsibilities to Reviewers 
Submission Reviews 

3. Rankings and Selections 

Topic Rankings 



40 



Submission Selections 




4. Announcements and Debriefings 

Bulletin Board Upload 

Posting Notifications of Selections 

1. Initiation 

Call for Papers and Intent to Submit 

After the conference committee announces the call for papers, authors are 
invited to fill in an intent-to-submit form that, when submitted to the server 
executes a CGI script that assigns a unique paper ID number to the prospective 
author and sends mail to the author with this number as well as instructions for 
submitting a paper to the conference. This information is also used early in the 
process by the conference committee to assign papers "to reviewers based on the 
proposed paper title and topic. 

Submissions Upload f 
The authors write their papers, mark them up in HTML under the guidelines of 
the conference, and archive their papers into single file archives using tar, PKZIP, 
or the equivalent program on their platform. The papers.are uploaded as binary 
files to the system via File Transfer Protocol (FTP) and are processed when a CGI 
script is submitted. The author connects to the conference FTP server via an FTP 
client and puts the submitted file into the specified incoming directory. The 
incoming directory is write-only and allows authors to write files only but not to 
read other uploaded unprocessed submissions as some submissions may contain 
proprietary information not yet released to the general public. An author then 
registers the submission with an HTML form in which the user enters information 
about the paper (submission ID number, title, conference topic, related 
keywords, uploaded file name) and the authors (name, E-mail address, mailing 
address, telephone, etc.). When the form is submitted to the server, the 
corresponding CGI script validates the information and if valid will perform the 
following steps: 

1 . Move the uploaded file from the FTP incoming directory to the processed area. 

2. Insert the author's profile (name, E-mail address, paper title, conference topic) 
into the submissions database table. 

3. Display a confirmation message as HTML output displayed on the client's WWW 
browser. 

4. E-mail the confirmation back to the author's specified E-mail address for the 
author's records. 

Problems with the Submissions Process 

These few steps introduced the first area of problems in that the instructions 
were not sufficiently clear and as a result authors inadvertently uploaded 
incomplete or corrupted submissions into the system. The first problem occurred 
during the transmission of the author's submission file to the server via FTP. One 
author was unable to connect to the FTP server because of an impossibly slow 



band-width transmission between the United States and the author's local 
network connection in Europe. This author had to UUENCODE the submission 
(i.e., convert the binary file to ASCII) and use E-mail to submit it. Several other 
authors uploaded their submissions (probably from a Macintosh or PC) and 

5 transferred the Tiles using FTP in ASCII mode, a process that translated carriage 
return and line-feed characters, resulting in a corrupted archive. Some authors 
submitted a multifile HTML document created using LaTeX2HTML [9], which 
made it difficult for a reviewer to view' or print the entire paper quickly. Other 
authors submitted multiple files (no archive) where it was not apparent which 

10 files belonged together. Furthermore, some of the submissions did not comply to 
the HTML 2.0 standard, so some browsers viewed them differently and other 
submissions had links to files that worked on the author's Web server, MS- 
Windows for example, but did not work under the UNIX server hosting the EMS, 
which has a case-sensitive file system and the file referenced as the HTML inline 

15 image in <IMG SRC="figurel.gif> differs from the file referenced in <IMG 
SRC="FIGURE1.GIF">. 

The next problem occurred when submissions were uploaded to the system 
correctly but were left unprocessed when the author neglected to submit the 
final HTML form to acknowledge upload, trigger the CGI script to enter the 

20 submission information into the database, and move the submission into the 
processed area for reviewing to commence. Without this final step the 
submission does not exist in the eyes of the reviewer. There needs to be a better 
mechanism that integrates safe and anonymous FTP with the WWVtto upload 
files, where authors enter file names on their local computer systems via a file 

25 dialog and the WWW browser uploads the files to the server, ft appears that 

Netscape is addressing this problem with its new Netscape Navigator 2.0 browser 
with the added "file" input type for HTTP file upload, but it will not be supported 
from all browsers and a solution across all browsers must be found. 
Some of these problems resulted from the conference committee not anticipating 

30 ALL the possible ways to upload a file, multiple file formats, HTML layouts, etc., 
while others resulted from authors not following the instructions. With an 
electronic layer between the reviewers, authors, and conference committee, it is 
not always clear to provide all information necessary for every situation 
especially when people may make assumptions of what is expected of them and 

35 others. A simple user interface and a clear set of instructions are best with 

extenuating circumstances handled on a case-by-case basis with correspondence 
between the author and conference committee. The author's interface to the 
EMS has changed little between the Fourth and Fifth WWW Conferences, but the 
author's instructions and the underlying CGI scripts have changed a great deal. 

40 The original instructions for WWW4 were about two printed pages in length 
while the revised instructions for WWW5 amounts to five printed pages outlining 
each step with troubleshooting information. With each iteration of the WWW 
Conference Series and the growth of the underlying WWW infrastructure (HTTP, 



HTML servers, and browsers), all of these problems are being handled quickly as 
we learn how to exploit this ever-changing medium. 
2. Submission Review 

^Z^^^m to preview the information entered about the 
LbrnS and the paper itself before even assigning it to reviewers. For 

l ££n£E by X m ging the submissions for a particular reviewer or 



30 



35 



40 



two reviewers 
Submission Reviews 



nn™^ are assigned reviewers will have access to these papers and be 

afeto'en^ 

a paper is selected, an evaluation form is filled in with ttejnform bon 
thLt nansr The reviewer grades the paper according to several cntena 

(Sevance XZ. cSne'ss, and quality) on a scale from 1 to i where 1 ,s 

Tt™:* the scores. P Th P e committee 

examine all reviews for a given paper to ensure that the scores between iwo 
revieTer^ do nTdiffer by more than one point in which case he committee 
must confer with the reviewers to re-evaluate their dec.s.ons and bridge the gap 



^X^Zents and modifications to authors' <«™^^ 
stored within the Oracle database assignments table bu p \^^ c f 
owing to Oracle's limitations on string data types with. n SQL stents. A 
VARCHAR2 datatype (a variable character type) can r not / n x n c f/ h ^° ers , 
characters, and the reviewers entering long reviews (>. ?^ J*^g 
resulted in some comments being truncated The ^^S^« the^ 
comments in external ASCII files that are outs.de the Oracle database but 
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accessible from the EMS's forms interface as well as directly readable from the 
Stem within the database the file name of the comments is stored and a 
S win the form triggers the CGI program to read the contents of the 
file and insert this into the form's TEXTAREA field for editing. . 
5 3. Rankings and Selections 

KScoring each paper there is another step to ran, each paper within 
a oiven topic to help compare the papers and identify the best or poorest papers, 
lor example there were 18 papers submitted to WWW4 belonging to the 

0 AuZa^i'n^nrnents topic. Of these 18 papers only four W« *«e 

Za^a 1 a xA/^rP rPiPcted so if they were ordered from one to 18, then tne 
toX ™ d "avt be^sS once the ranking was denned but ranking the 
n»nSv more difficult than grading them. It requires not only deciding whether 
fpape s goodS but aiso how it rates to all other papers within the topic. 

1 For tooic such as Charging and Payment Protocols with only three _ 
submStons it may not be difficult, but for the Authoring Environments topic t 
wouW take too much «me. Therefore, this recommended step in the peer review 
process is not always completed. 

and topic rankings, the committee decides whether to accept or reject each 
submission. 

4. Announcements and Debriefings 

,5 ^Z^^ons for the committee to generate a final report of the 
dedstons to accept and reject papers based on the evaluations. The report is 
^^f^f^fJs^M^ copied into the Bulletin Board area 
andTnkecl I from the main Bulletin Board. From there all reviewers and committee 
members can examine this report to determine whether any last minute re- 

30 evaluation is needed for any particular submission. 
Posting Notifications of Selections 

This step provides instructions for the conference committee to gather the 
submission information and evaluations, compose debriefing letters of 
loS or rejection to the appropriate authors with the reasons for reject™ 

35 or the needed modifications for a final revision of accepted papers, and 
electronically mail out the letters to the E-mail addresses of the designated 
authors. The authors of accepted papers are notified to submit final revision of 
their papers, which are uploaded in the same manner as the original paper 
through FTP followed by submitting a form to ingest it. 

40 Each process step has a deadline and when the selections are mailed out and the 
final p'ape * ptoSded to the conference se^er, then the peer review process £ 
officially over Then the conference committee query the databases and use the 
report tools to generate various text or graphics reports on the submissions and 



evaluations for analyzing ^ffiS^ Z nThaSpic^ oy 
conference. For example, some ™"™= n ^ £ m t the toplc may not be 
any or just a few a"*ors^s ^me art s mau P f 
relevant to the community or the .wording on . ^ seleded by 

from the WWW5 topics ' 

CGI and the Underlying Database 

, . . nh .,„ ,, , -multifunction" form that interfaces with a 
The heart of the user interface is ^a rnuraru fom prov|des a 

CGI program for accessing th \ d ^°f^ h " C ooeration s on a particular database 
single interface to perform multiple ^^"^"asof the database table 
table or view. The layout of the output and "rnals database 
are embedded within the HTM . form ^^^cg™ se^es all of the 

SbS^^ 

p^desTe - basic database operanons: 

operas searches for al. matches where all specified conditions are true 
(e.g., submission Jd = "14" <mduser_.d - "smnh ). 

° r " 5Mr ™s operation searches for any matches that match the specified coitions 

(e.g., submission^ = "117" or userjd = "jones ). 
Key-Search . . , ubmiss i 0 n id for the submissions 

Entering the PJ^^^^Tfa mat record and automatically fills 
table or userjd for the users tao >'> , corrcspon(J ing to the fields in 



record 

Insert 



0 



tn added into the appropriate database. Unique 

ffil » a « the same subm,sion ro number 

Update . . r _ rnrd should first be filled in from a 

request is submitted to save the changes to the database. 

Dek,e Delete removes a record or 

values. In most cases a user will search for a ^^"^^ als0 possib le is 



10 



15 



20 



form I, is high* recommended that users preview any record before deieting ,< in 
ca™,he wrofg record is accidentally chosen. 
Clear „ ^„»b«e operation, but it clears all information from the data 

2£ inf c^em " returns an empty form from which to enter a 

new search. operations. A sample database 

Each form ^.^^°^"Jg%£££U* HTML equivalent 
query form is illustrated in Figure 1 * fle | ds specifying what 
shown in Rgure 2 Hidden wrthn the HJMLtormw^ P ^ |g 
database is used (Oracle in thi s ca e, vvn ^ ^ ]dentif ^ the databa se 
the form name, primary tab e a ° mserted int0 ^ form with 
table. The column names of the da abas taDie are 

the NAME tag using the synttx f'"™.™™-™^* id column name, which 
fawenamet.opf/ona/ flags], for examp e ^ desl g nate d with the 

follows the "Paper ID #" text in the fo m, is a characte typ y .^.^ 
V data type from the ase tab le and^ ^ 

that it is a primary key (p) of the table ana m ^ CGI 

modifying the database (m). This form k * s a ' ' the cont ext. In 
program, which fills in the ?PPropna« >^ues depending | on form 
me case of an INPUT element, between the 

and the value of the "olumn at the bottom 

^a^^SST^ into the OPTION element that 
matches the value in the database. 
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} 



Figure 1: Sample query form for submissions 

<HTML> 

<HEAD> ___ 
<TTTLE>WWW95: Manage Submissions Form</TITLE> 
5 <BASE HREF="http://.../manage_submissions/"> 

</HEAD> 
<BODY> 

<Hl>Manage Submissions Form</Hl> 
<H3>(<AHREF="Overview.html">Overview</A>/ 
10 <A HREF="Relevant_Data.html">Relevant Data</A> / 
<A HREF="Report_Tools.html">Report Tools</A>)</H3> 

<HR> J( . , .„ 

<FORM METHOD="POST' ACnON="/cgi-bin/dbtool.cgi > 

<H2>Ghoose An Operation </H2> 
15 <SELECT NAM E= "operation" > 

<OPTION VALUE="AND"> And-Search - After filling in column values 

<OPTTON VALUE="OR"> Or-Search - After filling in column values 

<OPTION VALUE="KEY"> Key-Search - After filling in paper ID # - 

<OPT10N VALUE="INSERT'> Create Submission - After filling in column values 
OQ <OPTION VALUE="UPDATE"> Update Submission - After changing appropriate values 

<OPTTON VALUE="DELETE"> Delete Submission - After filling in column values 

<OPTION VALUE= "CLEAR" > Clear Form - Clears all values 

</SELECT> <A HREF="Instructions.html#operation"><IMG SRC='7Images/hlp_button.gif 

ALT="?"></A> 
<jc <HR> 

<INPUT TYPE="submit" VALUE="Submit"> <INPUT TYPE="reset" VALUE="Reset"> 
<HR> 

<H2> Fill In Key Values: </H2> 

<INPUT TYPE="hidden" NAME="dbms" VALUE="oracle > 
30 <INPUTTYPE="hidden" NAME="dbms_table_name" VALUE="submissions > 
<INPUTTYPE="hidden" NAME="primary_table" VALUE="submissions"> 
<INPUTTYPE="hidden" NAME="form_name" VALUE="submissions_form.html' > 
Paper ID #: <INPUT name="submission_id.c.submissions.pm"><BR> 
Submission date: <INPUT name="submission_date.d.submissions"><BR> 

35 List Pf keywords (as on title page): <INPUT NAME="keywords.c.submissions"xBR> 

<BR> 

Type of Presentation: <INPUT TYPE="radio" NAME="type.c.submission.r" VALUE="P"> Technical 

40 < a iNp r LrTTYPE="radio" NAME= n type.c.submission.r" VALUE ="R"> State-of-the-Art Report 

Decision: <SELECT NAME="decision.c.submissions"> 
<OPTION VALUE="NULL">Not defined 
<OPTION VALUE="l">Accept 
45 <OPTION VALUE="2">Reject 

<OPTION VALUE="3">Propose as poster 

</SELECT> <A HREF="Instructions.html#decision"xIMG SRC="/Images/hlp_button.gif 
ALT="?"x/A> 
</FORM> 
50 </BODY> 
</HTML> 

Figure 2: HTML corresponding to the form in Figure 1 



"i 
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Searching the Database 

oorfnrmina an And-search operation without selecting any values in the form will 
re£m a II s ubmte^ tor WW/4 and 211 for WWW5). Entering the 
kXrd a ^"se" for a search operation will match 14 submissions from 
tteWW4 EMS. The SQL statement generated for th.s operation is the 

following.^ unjquesubmissionJd/ title from submissions 

where upper(keywords) like upper('%database% ); 
This SOL statement will extract the submissionjd and title fields for an entries 

sSZ^table such that the string "database" b«t^ 
the keyword field. The output of this operation, formatted within an HTML form, 
is displayed in Figure 3 below. 



Figure 3: Search list of database related submissions 
Selecting submission number 112 from the list above and pressing submit will 
query the database for that entry and fill in the form w.th the values of that 
entry as shown in Figure 4. 



Fiaure 4- Sample query form filled in for submission #112 
Going back to the blank query form in Figure 1 and selecting the keyword as 
"database" and the overall decision as "accepted" with a numeric : value of 1 will 
result in a list of two matches (submission numbers 112 and 282). ™s 
5 operation will build the following SQL statement for matching both conditions. 

select unique submissionjd, title from submissions 

where upper(,keywords) like upperC%database%') 

More advantdse^ches! for example, can query for any submissions with either VRML 
10 or J A VA in the list of keywords by selecting the Or-Search operation and en tenng , he 
string "VRML, JAVA" into the keywords field. The comma operator delimits a list ot 
keys or substnngs that are added to the search parameters. Any combination of such 
searches can be entered. 

Security and Authentication 

15 Security is handled using basic user authentication (httpd, htaccess htpasswd 
htgroup) [10] where authorized users are assigned user names and passwords , 
and must authenticate themselves with their user namejnd passwor d before : 
they are able to access protected forms or CGI scripts. Most users are designated 
as reviewers with limited access (update reviews, examine submissions, etc.), 

70 and some users are designated as administrators with access tc > the > entire 

system and the ability to create new users. The .htgroup access file defines these 
two types of user groups: www-admin for administrators and www-users for 
normal users (i.e., reviewers). The forms that require administrator-only access 
have the appropriate .htaccess file specifying that only users from the www-adm,n 

25 group have access to that form. When creating a user on the system, not only 
does the users database table have to be updated with the user's ID, full name, 
E-mail address, and telephone number, but a corresponding entry for the user 
name and password must be maintained for the .htpasswd and .htgroup files ror 
server authentication. The database query tool knows nothing of the password 

30 files, so within the Manage Users/Internal Staff form (a form to create users) is a 
hidden field that specifies an external CGI script to run after the database 
operation is executed, where in this case the same CGI input is passed to an 
update-user script that updates the server authentication files corresponding to 
the selected database operation (insert, update, or delete). 

35 Report Tools for Analyzing the Data 

As stated earlier, the report tools generate textual and graphical reports across 
the databases corresponding to the current level in the system's menu hierarchy. 
For example, the report tools at the Examination of Submissions menu provide 
reports on the submissions, and the report tools at the Users/Internal Staff 
40 Processes menu provide reports on the users of the system. Some examples of 
the output generated from the report tools are discussed in the Appendix . The 
report tools are divided into two classifications: standard report tools and ad hoc 
report tools. 
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15 



20 



25 



30 



Standard Report Tools 

, ^ ^«ie =r« cnprialized tools that generate pre-defined reports, an 
Standard report tools are specialize i too is « J me ta-information 

dynamically generated HTML documents. 
Ad Hoc Report Tools 

^r»^^ - r,s?»r one - 

Sons One such tabular report is that in wh.ch a user enters a SQL 
statement to query the database direct^ and e 

Web. 

Conclusion 

TraSs in a hypertext menu structure breaks down a complex set of 
prases into a manageable step-by-step list of operabons. 
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Appendix. Results of the WWW4 Peer Review 
Process 



The following figures were generated as graphical reports from the WWW4 EMS 
after the peer review process was officially completed, which was on October 9, 
1995 when the final versions of the papers were submitted. 
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5 Figure 5: Number of papers uploaded by submission dates 

The submission deadline was July 17, 1995. However, many papers were 
submitted late; in fact, nearly 45% of the papers were late. As many more 
papers are being submitted to each WWW Conference, there will be a need for a 
stricter policy on late papers with an absolute deadline where only exempted 

10 papers are permitted a late submission. Further analysis of the data shows that a 
greater percentage of the earlier submitted papers were accepted (46%) as 
opposed to the late papers (18%) since the reviewers had more time to review 
them. 
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Figure 6: Number of papers per conference topic 

The subjects covered by the papers were diverse with a reasonable distribution 
as shown in Figure 6 and is also in the following table: 



5 1. Authoring Environments 18 

2. Charging and Payment Protocols 3 

3. Commercial Use 5 

4. Computer-Based Training and Teaching 17 

5. Consistency, Integrity and Security 6 

1 0 6. Design Techniques for Web Applications 11 

7. Information Representation & Modeling 9 

8. Integrating Object-Oriented or Relational Databases with W3 12 

9. Intelligent Search and Data Mining 6 

10. Knowledge Representation in W3 3 
15 11. Modeling Web Dynamics 2 

12. New Applications 13 

13. New Experimental, Commercial & Educational Systems 20 

14. Other 12 

15. Protocol Evolution and Extensions 10 
20 16. Resource Discovery 7 

17. Software for W3 Applications 8 

18. Time, Event Management & Monitoring 1 

19. Tools and Browsers 14 

20. Tuning, Benchmarking & Performance 3 



21. User & Application Interfaces 

22. Virtual Reality in W3 

23. Not Specified 
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Figure 7: Number of reviews assigned to each reviewer 

There were 28 reviewers for almost 200 papers with at least two reviewers for 

each paper, so many of them had to review over 20 papers apiece. 
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paper suitability grade 

Figure 8: Counts of overall paper suitability score for all reviews 
The grading of all the papers was fair with the expected bell-shaped curve with 
many papers receiving an average score (3) and fewer papers having a poor (1) 
or outstanding (5) score. 
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Figure 9: Count of relevance score for al! reviews 

The relevance grade asked the question of whether a paper works toward the 
goals of the conference where 1 is unrelated, 2 is somewhat related, 3 is of 
moderate importance, 4 is very important, and 5 is a topic of immediate 
importance to the Web. An interesting aspect of this graph is that most of the 
papers submitted were veiy important and relevant to both the WWW and the 
goals of the conference. 



WWW95 Paper Submissions Acceptance Report 




Figure 10: Number of accepted and rejected papers 
An abundance of papers was submitted to the conference, and many good 
papers were rejected because only a select few can actually be presented at the 
5 conference. Only 57 papers (or 29%) were accepted while 141 were rejected. 
Many papers were alternatively presented in the Poster sessions, and for other 
authors the committee recommended they resubmit their papers at the next 
conference with the suggested changes. 
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Introduction 

We began the Conservation Ecology Project in 1993 to address, through the dissemination 
and multi-media potential of the Internet, issues of cost, speed, and access to research 
findings in a rapidly changing and growing discipline. We realised that electronic 
preparation of manuscripts and the pervasive use of the Internet for communication, 
particularly within the scientific community, made it possible to conduct the entire peer- 
review process on-line. 

The Conservation Ecology Project capitalizes on this shift in the medium used for scientific 
communication from paper to electronic. It does so by automating the entire process of 
running a journal from submission through editing and review, to publication and 
distribution. That automation software has served as the editorial office for the on-line 
journal Conservation Ecology for over a year. 

Brief history of the Project 

We made the decision to publish on-line back in 1993. At that time, libraries were engaged 
in active triage to cut their expenditures and turnaround times were getting longer as 
journals accumulated accepted papers and deliberated over the tradeoffs between adding 
. issues and pressure to keep subscription costs low. In 1993, on-line journals were either 
solely text or required the reader to possess an advanced computer degree to decipher the 
articles. Email, ftp, and gopher were the dominant interfaces to the internet, and the World 
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Wide Web was just starting to emerge. In 1993, many authors, editors, reviewers, and 
readers were limited to email access to the internet 

Goals of the Conservation Ecology Project 

The key goals in publishing on-line were: 

sto^ Ao-turnaround time for reviewing and publishing manuscripts 

. accessibility: ^^.^ _ managers> NG0 - S , policy makers 

o world wide 

o platform independent 

benefits from the move "on-line". 

We've now published 3 issues of Conservation Ecology using the = software Ove. ■ P«** 
journal and dealing with the consequences of working in a new medium. 

The Submission System 

To give you a feel for how the software works, I'd like to "track" a manuscript submitted to 
Conservation Ecology through to publication. 

Manuscript tracking: Submission 

Authors play a key role in submission to Conservation Ecology. At present articles must be- 
fubSed by email However, the web upload feature shown below which can be used to 
SSS^T^ article (a highly simplified kind of article) will soon be amended for 

all articles. 
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Ail journals have formatting guidelines. Unlike other journals where! formattmg focuses 

\u „ a i., vied *n/1? and language and citation preferences, formatting tor 
SSS^^^Kring the manuscript for automated handling by our 
IZZ Z^Z L Changes from the familiar formats of prim journals were required 
the end to satisfy the Conservation Ecology system requirements. 
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. Each text figure, table, or appendix must be submitted as a separate file. This is 
fSS'plS journal; that require that these be each submitted on a separate page at 

. ^"m^te^ed to identify parts of the article (e.g., 

text appendices), and author information (e.g., name, address). We provide a text 
template for authors to copy into their word processing package and use. 

. A^List of Attachments" is required. This list includes the caption for each table, 
figur and appendix (similar to print journal requirements that captions for figures be 
S seperaSy in a "List of Figures"). The software then links the caption to the 
a^ro^ffle This split between caption and content allows readers to peruse the 
figure caption while waiting for the related graphic to load. 

The automated submission module 

The above formatting provisions allow the software to: 

. assign a manuscript number and start a database entry for the article; 

. mount an HTML version of the complete article (text, figures, tables, and appendices) 

wZut aumor information and acknowledgments at a password-protected review site 

(we^run double blind review); ' 
. create ASCII versions of acknowledgments, abstract, keywords, author, and title 

information for use in messages to authors, editors, and reviewers; 
. initiate the manuscript tracking software, send an acknowledgment of receipt of the 

article to the author, and notify the EIC of new submission. 

Manuscript tracking: the editor-in-chief 

Having been notified of the new submission by email from the system the editor-in -chief 
(EIC) consults the new manuscript on-line using the EIC web page which includes links to 
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:S t^SSppiy more .nan one po.en.ial edUor we elirntnate the 
potential for lime delay should the firs, editor contacted not be able .0 handle the 

manuscript at that time. 
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Manuscript tracking and peer review: The subject editor 

The subject editor (SE) consults the MS on line and if appropriate for review, the SE sends 
to the system: 

. a list of expertise categories (e.g. small mammal movement), 
. the number of reviews required from each of these, and 
. a prioritized list of potential reviewers for each category. 

The expertise categories allow the subject editors to flag reviewers to attend to V*™^ 
aspects of the article according to their expertise. In the example below, small mammal 
movement" has been identified by the subject editor as an area requiring special attention . 
for this manuscript. As for the Editor-in-Chiefs specification of potential subject editors, 
the subject editors is required to recommend 2-3 times the number of potential reviewers as 
is ultimately required. 
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The software now contacts the required number of reviewers in each category, starting from 
the top of the list for that category. The software keeps track of: 

. who has agreed to review and sends them review instructions, 
. who has not yet responded to the request and nags them, 

. wh^ catVry requires another reviewer to be contacted ^fT^Z^ 
contacted earlier has declined (note that the reasons for declining, if supplied, are 

passed on the to SE), and .... 
. when the SE needs to be contacted for additional potential reviewers. 

All this information is captured in the database and can be viewed, by the Editor-in-Chief 
r^r* X, and subject editors, through reports such as the manuscript history shown 

below. 
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Manuscript tracking: the nagging system 

At the heart of the manuscript tracking and peer review software is the timing and 
"nagging" system. This system comprises: 

. time frames (how long each activity should take), 

. reminder and nagging loops (e.g. prompting a reviewer that their review is due in 3 



days, telling them that it is due today, telling them that it was due last week), and 
• chains of command - who should be contacted if particular reminders don't elicit a 
response (if a reviewer doesn't submit a review after two late reminders, then the SE 
is notified, appraised of the history, and asked to intervene). 
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Since these processes are automated, they work at night, on weekends, and on holidays 
unless you tell them not to (we adjust deadlines over the Christmas/year end period, for 
example). 

You can see from the list of scheduled reminders below, how the system is set up to nag 
people unless they deliver reviews, recommendations, and decisions on time. 
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Manuscript tracking and peer review: copyediting 

Once the reviews are submitted, the subject editor has made a recommendation, and the 
Editor-in-Chief has accepted the manuscript, the software automatically 



creates the "copy edit" draft, 

places this in a password protected ftp site, and 



• notifies the copy editor. 

The copy editor works on an html version of the manuscript. Information such as the title, 
author, and keywords that is used in several locations is edited by the copyeditor using a 
web screen so 'that the database is updated and the different presentations of this information 
are consistent (e.g. Table of Contents, abstract file, metadata). 
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Manuscript tracking and peer review: publication 

Once copyedited, the final version is created; volume,, issue, and article numbers assigned; 
and the authors contacted to approve the on-line proofs (still at the password protected 
review site); Upon approval by the authors, the article is mounted in a pabliely-accessible 
article directory, and the table of contents of the Issue-in-progress is updated At regular 
intervals (at present biannually) a volume is declared complete and subscribers are 
automatically notified of its contents by email. 



Research 

Local vs. Landscape Effects of Woody Field 
Borders as Barriers to Crop Pest Movement 




Did we meet our objectives? 
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SPEED 

With the automated peer review system, all clerical steps (e.g. acknowledgement of receipt 
of a manuscript, nagging and prompting messages) are substantially faster. The machine 
works on weekends and at night. The sending out of prompts, reminders, and 
acknowledgements is not constrained to "normal business hours". In an international 
workine environment, immediate response to incoming messages and commands can save 
severafdays as delays due to incongruent time zones are avoided. The software has no 
"time zone". 

It is not clear whether we have realized any gains where people are required to make 
decisions -e.g. reviewing and editing - these still take time. However, we will be amending 
the time frames for all steps in the process shortly to see if we can speed people up. Such an 
experiment is achieved simply by setting new values for the time steps for each process. 
Should the experiment prove unsuccessful, or successful only in certain areas, then we can 
simply reset the relevant values to the original settings. 

Copy editing is much faster as the bulk of the formatting is done automatically. Publishing 
is faster both do to automated formatting and table of contents creating as well as the fact 
that articles are published as they are copyedited in an "Issue-in-Progress". 

COSTS 

It is difficult to quantify the savings realized, as we are still in a development with regard to 
the software. However, it is clear that we are realizing savings through the software even 
before development is complete. 

Database entry is minimized as the authors and the software do the bulk of this. All standard 
clerical steps - acknowledgements, nagging, prompt - are automated. No paid staff are 
required to do these tasks. All correspondance is conducted by email, and is therefore free. 
Formatting, for both copyediting and publication, are done by software. Printing and 
distribution costs eliminated as Conservation Ecology is published only on line. _ 

We have incurred some additional costs. We require a 1/2 time production editor to confirm 
conversion to HTML was successful for submitted articles and to correspond with authors. 
As we run our own server (on a Sun SparcStation 4), we employ a 1/4 time system 
administrator and have incurred hardware costs. 

Unanticipated benefits 

There are numerous benefits to the software that we had not anticipated when we set out to 
create an automated editorial office. One such benefit is archiving. All versions of the 
manuscript are archived by the software (including all author and title information which 
may change between versions). All correspondance to and from the software is logged 
automatically. Correspondance outside of the system can be added to the log by the editor- 
in-chief, managing editor, and subject editors. Lastly, we archive a complete record of the 
timing of all events associated with a particular manuscript. 

Meta data was not a concern for us when we began this project in 1993. However, having 



all the necessary information stored in an electronic database makes it easy to build 
metadata and apply to published article as part of formatting template. It would also be a 
simple matter to change or update the format of that metadata as standards evolve. 

Lastly, putting the entire process of running a journal on-line removes the requirement to 
have a centralized editorial office. Access to information is available on a platform 
independent basis over the web. The degree of access to the database is determined by 
username and password access codes that tell the software what level of information to 
display (reviewers never see the authors' names when they review of manuscript, whereas 
the subject editor does) and whether the user has read only or write access to the database 
and to what sections. 

The distributed nature of access to the system allows journal-participants to contribute to the 
peer review process from anywhere at anytime. They do not have to have access to their PC 
on their desktop to do their job for Conservation Ecology. Journal participants don't have to 
maintain separate files on the manuscripts on which they are working. By providing them 
with simple web-tools to do keep, track of the manuscripts on which they are working, we 
facilitate and reinforce the updating of the journal database files. 

What next: Improving the software 

The software will have to be continually updated to keep pace with developments in 
authoring tools (e.g. word processor and graphics packages) and on-line publishing (e.g. 
.XML, mathML). We would like to move to an SGML base, rather than HTML, to give us 
the flexibility to output to a wide variety of formats (e.g. HTML, print,. CD Rom) and for 
better archiving. 

We would like to upgrade to a new database. We use a shareware database called mySQL 
which works, but is rudimentary and slow. We're looking into purchasing a commercial 
database and ammending the software to take advantage of it. 

By the end of this year we will have finished developing and testing this first version of the 
software. We will have the web interfaces built that allow authors, reviewers, and editors to 
interact with the journal software using the point and click environment of the web as well 
as using email. We will have processed about 100 manuscripts in 1998 with the software. 

So, we're at a point where we've got a working piece of software that handles the bulk of the 
clerical steps of running a journal on it's own, structures the review process with prompts 
and nagging to keep it on time, tracks all the transactions associated with a manuscript, and 
allows reviewers, editors, and authors to participate in the peer review process from around 
the world. 

What next: Generalizing the software 

We'd like to move the software forward to a generalized form. By generalized, we mean 
able to handle the peer review system for any journal, and also possibly for grant review 
and conference abstracts or proceedings. 

We'd like to make the software more robust and increase its capacity to handle any volume 
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of manuscripts. The current version of the software, largely due to the database, is limited in 
the number of manuscripts that it can handle efficiently. We're running around 100 a year 
now without a problem, but a volume of 1000 or more would be cumbersome to handle 
with the current system. 

Achieving generalization 

What will it take to make the software generalized? 

• A "configuration" interface that allows the software user to set the system up for their 
particular situation - including management structure, time settings for prompts and 
nagging, adjusting the templates for the format of the output and reports. 

• Complete documentation. 

• A help system - so that end user can install the software and troubleshoot. 

We had always hoped that the software and our experience would benefit more than just 
one journal. There are now a number of on-line journals using the web to "distribute" papers 
during review. Some journals even allow the authors to check the status of their manuscripts 
online. No journal that we know of has automated the entire peer review process and 
integrated it with online submission and publication. 

Conservation Ecology is interested in partnering with other societies and publishers to 
refine the software for our mutual benefit. The software works for our case. To make it 
work for other journals, we need input from other societies, and other publishers. 

We have already invested US$233,000 in Conservation Ecology, the new journal, and in the 
peer review software. We're interested in developing a consortium of society publishers to 
take the software forward. We would like the gains realized from electronic publication to 
extend beyond cutting the costs of printing and distribution from the budget We'd like to 
see other journals benefiting from the shift to the new medium and the opportunities that 
that shift presents. 

We have not yet defined the structure of such a consortium. Rather we are inviting others 
interested in realizing the full benefits on on-line publishing to join a discussion as to how 
best to achieve this. Please contact Shealagh Pope, the Project Coordinator for the 
Conservation Ecology Project, at sepo pe@consecol.org if you have any comments or 
questions. 
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Open Peer Review & Argumentation: 
Loosening the Paper Chains on Journals 

Tamara Sumner and Simon Buckingham Shum of the Knowledge Media Institute of 
the Open University, describe the design and implementation of the Journal of Interactive 
Media in Education, as well as some of the issues behind the approach that this journal 

takes to the process of scholarly review. 
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20 Summary 

The emergence of the internet and the World Wide Web (WWW) have potentially 
profound implications for scholarly practice, particularly in the submission, review, and 
publication of articles in journals. However to date, much of the impact of these new 
technologies on journals has been on digitising the products of journal publication; the 

25 scholarly processes involved in reviewing articles remain unchanged and unsupported. . 
We are using computer-supported collaborative argumentation (CSCA) tools, to rethink 
and redesign the process of scholarly debate at the heart of journal reviewing. This paper 
describes the design principles behind our approach, how they are currently being 
realised in the context of a specific new journal, and discusses some of the issues that this 

30 approach raises. 

Keywords: digital journals, argumentation, electronic publishing, hypermedia, 
WWW 



The shift from paper to digital media 



The emergence of the internet and electronic publishing have far reaching implications 
for the way in which knowledge is disseminated and sanctioned within scholarly 
communities, which since the Gutenberg revolution in printing has been shaped by the 
affordances of paper. Currently, we are witnessing the beginnings of a shift from paper to 
5 digital media in the submission, review and publication of scholarly work. In many 
journals, it is now standard procedure for reviews and submissions to be transmitted 
electronically. Sometimes, automated systems are in place for matching reviewers with 
submissions based on keyword analysis. In a few cases, the final version is even 
published electronically as a set of HTML files available on the WWW. Thus to date, this 
10 shift from paper to digital media has mainly affected the products of publication. Various 
documents are becoming digital and some activities are automated, but the actual process 
of scholarly work has not changed to a great extent. 

What remains completely unsupported is the intellectual 'meat 1 of scholarly 
publication, the review process itself. This remains a private affair involving a 
15 small number of reviewers, and in most paper-based journals, the gap between 
initial submission and article publication can be dragged out over a year. The 
potential of internet technologies for opening up and revitalising the scholarly 
debate process has yet to be realised. 

Argumentation: the heart of scholarly debate 

20 What is the nature of intellectual work in the peer review process? Reviewing a 
submission involves drawing on certain criteria (e.g. theoretical content; empirical 
content; presentation quality; appropriateness for the publication) to evaluate the quality 
of reasoning and evidence provided, to probe for weaknesses, acknowledge strengths, and 
question background assumptions. In effect, reviewing is an argumentative process where 

25 reviewers are engaging in an imaginary debate with distant authors who are not present to 
respond to their analysis. This paper-based review model has shortcomings in that 
questions go unanswered; confusions go unclarified; criticisms go undefended. The 
dynamic cut-and-thrust of debate normally found in face-to-face contexts such as 
workshops or conferences is not supported by the paper-based review processes, nor is it 

30. yet being realised in the new electronic media. 

We are currently rethinking the review process to use new technologies in order 
to recapture the best features of a dynamic scholarly debate. This rethinking is 
guided by existing research into hypertext-based, computer-supported 
collaborative argumentation. Argumentation research is concerned with 

35 developing notations and tools to facilitate public debate and negotiation. 
The remainder of this paper will begin by reviewing previous work in 
argumentation and distilling important design guidelines for creating 
argumentation tools. Next, we will describe how these guidelines are being 
followed in the context of a specific new, tool-supported journal publication. 

40 Finally, we discuss some of the issues which may be encountered in the new 
review model and its supporting technology. 

Computer-Supported Collaborative Argumentation (CSCA) 

CSC A research has focused on designing notations to support debate processes [1,3] , 
creating computational tools to support using the notations [4, 6, 9] and understanding the 
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organisational contexts and work practices necessary for such notations and tools to 
succeed [5, 1 1]. Key lessons have been learned from previous research that point to 
design guidelines in each of these three areas: 

• Lesson 1 : Avoid elaborate and rigid notations. Much research has focused on 
5 finding the "right" notation to support debate. Often, the resulting notations 

require people, to express their thoughts using elaborate sets of provided 
distinctions such as positions, issues, comments, pros, and cons. This, however, 
runs the risk of burdening people with excessive representational overhead by 
forcing them to categorise and commit their thoughts to rigid notations before 
10 they are ready. Empirical findings indicate that people are often unwilling and 

sometimes even unable to do this [1, 2, 10]. 

. Lesson 2: Computational tools must integrate argumentation with the artifacts 
being discussed. Early Csca approaches separated the argumentation from other 
artifacts (i.e., the papers and drawings being discussed). This separation hindered 
1 5 users from (i) quickly accessing relevant argumentation when it was most needed 

during problem-solving, and (ii) easily adding new argumentation. Later work has 
shown that tools must support users to bridge the separation and move seamlessly 
in both directions between the artifact and argumentation [4, 8]. 

• Lesson 3: Work practices must be redesigned so that argumentation is integral to 
20 the task being performed. Empirical studies indicate that people-often do not 

contribute to argumentation because it is perceived as extra work over and above 
what they are already required to do [5]. Successful argumentation approaches 
have redesigned work practices to make contributing to argumentation integral to 
the overall task being performed [11]. 

25 A CSCA environment for scholarly debate 

Our goal is to develop a CSCA environment for scholarly review of journal submissions 
which takes into account these three lessons. This environment is being developed to 
support the reviewing process of a new scholarly journal, the Journal for Interactive 
Media in Education (JIME). 
30 The JIME Web environment has been carefully designed in accordance with 
three design principles deriving from the three lessons introduced above. 

Principle 1: Begin with a simple argumentative notation 

The notation used to support reviewing in JIME has one primary class of contribution 
called comments. Optionally, debaters may choose to categorise comments as agreements 

35 (flagged as or disagreements (^r). The power of the Web means that reviewers may 
elect to include pointers to evidence in their comments; evidence can be either plain text 
(e.g., "See Conklin & Begeman, 1988") or a Web hyperlink direct to an online source. 
Thus, a body of secondary evidence may grow around an article under review, providing 
a valuable resource for both readers and authors. 

40 Principle 2: Integrate the publication with the argumentation 



Prior to publishing for open debate, JIME submissions are marked-up to include special 
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argumentation tags C**f ). Each major document section and the interactive 
demonstrations are prefaced with these navigation tags. Clicking on one takes the reader 
to the argumentation about that document section, in the Review Window. Conversely, 

5 from the Review Window, debaters can easily switch back to any part of the publication. 
Figure 1 shows the user interface for reading an article in the JIME Web environment. 
Reviewers can also download an Acrobat version since we believe that there is still a 
strong preference for being able to read a paper version. Thus, a reader/reviewer has two 
main 'work spaces' in JIME - the Article Window for reading, and the Reviews Window 

10 for making review comments, which are automatically opened and brought to the front as 
needed. . ^ , 
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Figure 1: Reading an article in JIME. In the lower left frame, the 
Comment icon takes the reader to the top of associated Review 
15 Window (see Figure 3 below). The printer icon allows the reader to 
download an Adobe Acrobat version for printing. In the main frame, 
the Comment icon takes the reader to the review comments about this 
figure and associated demonstration. The Demo icon takes the reader 
to a Macromedia Shockwave demo of the system (see Figure 2). 
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Figures 2-4 show screens for trying out a demonstration (Fig. 2), displaying the 
Review Debate about an article (Fig. 3), and reading Reviews about a particular 
section of the article (Fig. 4). The Review argumentation is added and 
manipulated using an environment which we have created by tailoring the NCSA 
HyperNews system [7]. 

Art Explorer: Interactive Demonstration 1 

'Seeing' 

This is an interactive extract on ' Seeing' , from Episode 2 of the Art Explorer (see main 
article for details). 

Try moving and modifying some of the elements in the painting. . . 




Figure 2: Authors describing a new system must submit 
demonstrations of it using an appropriate technology. The above 
example shows an interactive demo using the Macromedia 'Shockwave' 
plug-in. This allows readers and reviewers to interact with the system 
described in the article, as though they were students using the 
original application. 
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Netscape: JIME-Q1 Reuiem Debate 



Location : ihttp :/7www-jime .open .ac .uk / Reviews /get / jime-0 1 -reviews .html 



JtWE 



Dmbndae & Stntfbld: yiiyifta T*xtutg...An . Lamina t3 Multimedia 

V\ev S ecnon: Atetact • l • 2 Li - Li * 11 • Id - 'Li - Li • I * 11 * 11 " H 

- 4 • rtefe - Demo 1 • Demo 2. 



Submission: Varying the Texture: A Study of Ait, Learning &. 
Multimedia 

Authors: Durbrid* e 8e Stratfcld 

Editors: Simon Buddn&Um S\um & T^rni* Sumner 

The paper describes both i pirociU: multmedu title, An Explore*, ind in ippucacH torpids 

Cretans *iticwiaittl multim*iii in s*n*nl Th* ^enenl it^rc^K .idvo^tes that dasi^n fitsr bsgins 



Areas for De oate [Inline Depth: 1 2 3 All] [Outline Depth: 12 3 9 AH] 

^ Orig inality & Importance of Idea? 

Ori g inal analysis and an iraiovanVe , veil- motivated system 
^ Clarity of Goals 

Project g oals and, paper goal? clearly articulated 
^ Appro priateness of Methods 

More information on stud-leg needed 

^ Yea . more details on the studies needed ! 
Link desi g n decisions more closely to learner analysis 
^ CLariT v & Credibility of Results 

Has Art Explorer been used by students? 
No discussion of limitations . or comparison vith related vork 
Art Explorer best for distance learnin g? 
^ Q uality of Writin g 

Overall , vritin g clear 
^ 1. Introduction 

Claims not foHoved up later in article 
Ho reference to related vorX 
^ 2. Learner Heed 

What does "a ppro priate structure" mean? 
► 2.1 PerceptuaUConceptuai 



mm 



Figure 3: Overview of a review discussion in JIME. The example shows 
the outline view with headings for general categories of comment (e.g. 
Clarity of Goals; Credibility of Results), followed by headings which 
match the headings in the article. Section-sepcific comments are 
placed under these. 
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lai Netscape: 5.5 Episodic Design: The Ingredients of Art Eh iEa= 



Location: http://www-jime.open.-ac.uk/Reviews/get/jime-01-revievs/17.htrr * fr 



jytere? 'M5ffi?fes| Prey, t 'tEHj Ou*- 



^3.3 Episodic Design: The Ingredients of Art 
Explorer 



Base: fIME-0 1 Review Debate 
/toft -Wed, 31 Jul 1996 09:22:21 GMT 



I .ink ne eded between episode features and 
learner needs 

3.3 Episodic Desi g n: The In g redierus of Art Explorer 
Dttv/Fri, 09 Aug 1996 16:48:46 GMT 
JQivn. ] jme@o pen.ac.uX 



It vould be nice if the key aspects of each episode vere more closely tied back to ; 
the findings in the learner needs section. * ' 



f& Make explicit the implicit design criteria 

i^i Link needed between episode features and learner needs 
Itoto'Fn, 09 Aug 1996 16:50:51 GMT 
. film S . B uc kin g ham. Shum@ o pen. ac . uk 

I Definitely - it's implicit that AE is satisfying these criteria through its various 
! episodes/textures - this needs to be much more explicit 



E pisode 1 description inadequate 

A 3 3 KTiisoriir nftswnr Thg Imreil ientt nf Explorer 



Figure 4: Details of a Review discussion in JIME. The example shows 
review comments about a specific section in the article (3.3). 



5 Principle 3: Make argumentation integral to the review process 

The proposed review model for JIME submissions is shown in Figure 5. This model 
changes the role of participants in the process to directly involve generating and 
manipulating argumentation. Reviewers return their comments to the editor in the JIME 
argumentative format. The editor pulls together all the reviews to seed the argumentative 

10 debate. The publisher marks-up the publication and the initial argumentation to include 
the special cross-tool navigation tags. The article under review and the reviewers' initial 
comments are then published on the Web, and the review process moves into a phase of 
open peer review, in which authors, reviewers and readers can engage in debate. The 
editor then decides whether the article should be accepted, and formulates change 

15 requirements for the authors. We also wish to allow for the possibility that interesing 
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discussion threads may arise during the review process which could be distilled into 
commentaries for publication with the final article. 

Editor decides on whether to 
accept article, specifies changes, 
and considers whether to publish 

Editor verifies relevance Editor seeds review area other discussion threads as 

of article to journal ' with reviewers' comments commentaries 



Editor verifies revision 
& any related 
commentaries 



Authors submit article 
♦ demonstrations 



Reviewers structure their 
reviews using the JIME 
ar gum ent at io n s c hem e 



Open peer review - 
debate between readers, 
reviewers & authors 



, Final version 
I published — 
1 debate continues.. 



Publisher prepares 
final version 



Publisher marks up article Authors revise 

& review argumentation paper 
with navigation tags 

Figure 5: Lifecycle of a JIME article under review. 

5 Once the article is published, debate about it will continue. Wth conventional 
journals, it is only at this point that debate would begin, perhaps a year or more 
after initial submission. Even then, the debate is not tightly linked to the article, 
but located in other articles, often published in other journals, at substantially 

10 later times. In JIME, discussion can continue directly linked ta the published 
article, or via external links to related articles at other sites. 
To summarise our design philosophy: 

• JIME is not simply making the conventional media of paper-based articles 
available on the Net, but addressing the interactive dimensions which are the 

1 5 essence of the new media; 

• JIME is not simply adding an e-mail listserver to allow peripheral discussion 
about published articles, but developing a journal review environment in which 
submitted articles and open peer review are tightly integrated with each other, and 
central to the journal's operation. 

20 Discussion 

This project is still in its early design stages and the details of the current interface will 
undoubtedly change as it undergoes further implementation and user testing. However, 
we feel it is important to discuss work such as this in its early stages in order to 
encourage more participation and dialogue between scholars concerning future 
25 publishing practices. We believe the review model presented here offers several benefits 
for opening up and revitalising scholarly debate. However, there remain many open 
issues to be resolved. 

Some issues relate to the user interface for supporting collaborative, Web-based 
argumentation. For instance, the Web introduces new concurrency challenges 
30 regarding timely updating of reviewers' screens with new contributions to the 
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discussion. Additionally, Web-based systems offer impoverished degrees of 
interactivity compared to what we now expect from direct manipulation 
interfaces, limiting our ability to take advantage of new developments in • 
graphical argumentation tools. 

Another issue raised by our proposed model is the perceived risk which authors 
may feel when exposing publications to large audiences at the review stage. 
Submissions will be critiqued by many more reviewers than is currently possible 
in the conventional review process, and this will take place in a public rather 
than private forum. The willingness of authors to do this depends greatly upon 
the professionalism and netiquette exhibited by reviewers. The philosophy 
behind this model is that perceived risks of this sort will be outweighed by the 
benefit to authors of quicker, more extensive, and more relevant feedback, 
resulting ultimately in higher quality publications. 

To conclude this brief article, we should not be surprised that electronic journals 
meet with resistance from some quarters, since they force to the surface huge • 
issues which entail rethinking how scholarly knowledge is disseminated and 
sanctioned. This is the time to reflect radically and creatively on the 
'papyrocentric' scholarly practices which have until recently been taken for 
granted. JIME is pushing the boundaries of electronic journals one step further 
with its Web-based open peer review. This seems to us the logical direction in 
which to take journals. We await your reactions with interest. 
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